Transform Data Using Azure Synapse Pipelines – Transform, Manage, and Prepare Data-2

  1. Log in to the Azure portal at https://portal.azure.com ➢ navigate to the Azure Synapse Analytics workspace you created in Exercise 3.3 ➢ on the Overview blade, click the Open link in the Open Synapse Studio tile ➢ select the Manage hub ➢ select SQL Pools from the menu list ➢ start the dedicated SQL pool ➢ select the Data hub ➢ expand the SQL database ➢ expand the dedicated SQL pool ➢ expand the Tables folder ➢ and then confirm that the dimension tables you created and populated in Exercise 4.13 are present. Refer to Figure 4.35 for the list.
  2. Select the Develop hub ➢ hover over SQL Scripts ➢ click the ellipse (…) ➢ select New SQL Script ➢ select your dedicated SQL pool from the Connect To drop‐down list box ➢ and then remove the [brainwaves].[FactREADING] table by adding and executing the following SQL command to the SQL script window:
    DROP TABLE [brainwaves].[FactREADING]
  3. Create a stored procedure named uspCreateAndPopulateFactReading (the syntax is in the directory Chapter05/Ch05Ex01 on GitHub here at https://github.com/benperk/ADE) ➢ place the SQL syntax into the SQL script window ➢ click the Run button ➢ name the SQL script (I used Ch05Ex01) ➢ and then click the Commit button to save the SQL script to the GitHub repository you linked to in Exercise 3.6.
  4. Select the Data hub ➢ select the Workspace tab ➢ expand the SQL Database folder ➢ expand your dedicated SQL pool ➢ expand the Programmability folder ➢ expand the Stored Procedures folder (you should see the just created stored procedure) ➢ right‐click the stored procedure ➢ select Add to Pipeline ➢ select Existing Pipeline ➢ click the pipeline you created in step 8 of Exercise 4.13 (IngestTmpReading) ➢ click the Add button ➢ on the General tab enter a name for the SQL pool Stored Procedure activity (I used CreatePopulateFactReading) ➢ and then select the Setting tab and review the configuration.
  5. Notice the small rectangle on the right middle of the MoveToTmpReading Data Flow activity ➢ click, hold, and drag it over to the CreatePopulateFactReading SQL pool Stored Procedure activity ➢ release the mouse button ➢ and then click Commit to save your changes. The pipeline should resemble Figure 5.3.

FIGURE 5.3 Azure Synapse Analytics—transformating Brainjammer brain waves

  1. Click the Validate button ➢ click the Publish button ➢ review the proposed changes ➢ click OK ➢ click the Add Trigger button ➢ select Trigger Now ➢ click OK ➢ select the Monitor hub ➢ select the Pipeline Runs menu item ➢ and then view the progress. You will see something similar to Figure 5.4. Note that you might need to click the Refresh button to see the changes.
  2. Execute the following queries, and then confirm the data was ingested, transformed, and moved as expected:

FIGURE 5.4 Azure Synapse Analytics—monitoring Brainjammer brain wave transformations
SELECT COUNT(*) AS [COUNT] FROM [brainwaves].[FactREADING]

  1. Pause the dedicated SQL pool and consider renaming the pipeline with a more appropriate name (for example, IngestTransformBrainwaveReadings).

Raymond Gallardo

Learn More →

Leave a Reply

Your email address will not be published. Required fields are marked *