IBM InfoSphere DataStage Fundamentals

Shared containers in DataStage are created separately and stored in the Repository for reuse across multiple jobs. Local containers exist only within the specific job and are not stored separately in the Repository.

In the Transformer stage, execution order is: (1) Stage variables are initialized, (2) Constraints are evaluated to filter rows, (3) Derivations are computed for output columns. This order ensures derivations can use constraint results and stage variables.

The Join stage requires all inputs to be sorted by the join key columns (requirement for merge join operation), and it supports inner, left outer, and right outer join types. Both statements are true about Join stage behavior.

The Filter stage in DataStage allows field names to be specified using job parameters, making field mappings dynamic and reusable across jobs. The Funnel stage combines multiple inputs, not parameterized field mapping. The Transformer stage uses expressions but job parameters for field names is a specific capability of the Filter stage.

In the Sequential File Stage, the 'Output' Reject Mode writes rejected records to a reject link for later handling. 'Fail' stops the job entirely on rejection, 'Drop' silently discards rejected records, and 'Continue' processes remaining records but doesn't write rejects anywhere.

Keyless partitioning methods don't require a key field to distribute data. Round Robin, Random, and Entire are keyless methods. Modulus and Hash are both key-based partitioning methods that require a specific key field to calculate partition assignments, so neither qualifies as keyless.

The Job Activity stage specifies and calls a DataStage server or parallel job within a sequence. It's different from the Command Activity stage, which executes operating system commands. The Job Activity is fundamental for orchestrating multi-job workflows in DataStage sequences.

IConv (Input Conversion) converts external string representations to internal storage formats, like converting a date string to internal date format. OConv (Output Conversion) does the reverse - converts from internal to external display format. Both are DataStage BASIC functions for data conversion.

DataStage parallel processing uses two main types: Pipeline parallelism, where multiple stages run simultaneously with data flowing between them, and Partition parallelism, where data is split across multiple nodes for simultaneous processing. Both techniques improve throughput by leveraging available resources.

DataStage parallel environments fall into two main categories: SMP (Symmetric Multiprocessing), where multiple processors share memory, and MPP (Massively Parallel Processing) or Cluster architectures, where multiple nodes have their own memory and communicate over a network. These represent the two fundamental hardware architectures for parallel processing.

DataStage reads the configuration file on startup to understand system resources like number of nodes, available disk, memory, and scratch space. The configuration file defines the processing topology and resource pools. This file is essential for proper parallel job execution and resource allocation.

Stage variables in parallel Transformer stages can execute functions once per record for row-level calculations, once per run for initialization, and identify the first row of an input group using @INROWNUM or other techniques. This versatility makes stage variables powerful for complex transformation logic requiring different execution scopes.

DataStage supports exporting metadata including field definitions for sequential files through table definitions. You can export, import, and reuse metadata across jobs using the Repository or by exporting to .ds files. The statement claiming we can't export metadata is false - metadata management is a core DataStage capability.

The Aggregator stage produces double precision floating point numbers by default for aggregated values like SUM, AVG, MIN, MAX. Using double precision prevents rounding errors and maintains accuracy for decimal calculations. Single precision or integer would lose precision for most aggregations involving division or decimals.

Round Robin partitioning yields the most even distribution by systematically cycling records through each partition (1 to partition A, 2 to B, 3 to C, repeat). This guarantees even distribution regardless of data characteristics. Hash can create skew if key distribution is uneven, Random may cluster, and Entire duplicates all data to every partition.

Job parameters in DataStage are flexible variables that can be referenced throughout your job design. They're commonly used in constraints for filtering data, in derivations for data transformations, and for dynamically specifying file paths or directory names. This makes jobs reusable across different environments.

The Annotation stage in DataStage is specifically designed for documentation purposes. It allows you to add descriptive text, comments, and metadata to your job design without affecting the data flow. This helps other developers understand the job's purpose and logic.

DataStage Director is the runtime and monitoring component - it schedules, runs, and monitors jobs but does not compile them. Compilation is done in the Designer before the job can be executed in Director. This separation of concerns allows non-developers to run jobs without modifying them.

DataStage Director is specifically designed for job scheduling, execution, and monitoring. Administrator manages server configuration and security, while Designer is for job development. Director provides the interface to schedule jobs, monitor their progress, and view logs.

Hash partitioning is optimal for join operations because it distributes rows with the same join key to the same partition. This ensures matching records from both inputs are colocated, minimizing data movement during the join. Round robin distributes randomly, Entire copies everything to all partitions, and Auto may not choose the best method.