Skip to content

Data Source#

The DataSource decorator is used to define a function that returns data to be used by a widget, or a set of widgets. The function can be used to retrieve data from a database, a file, or any other source.

decorator classmethod #

decorator(
    _func: Optional[Func] = None,
    *,
    storage_type: Optional[StorageType] = None
) -> Any

Decorator to store data in the Dashboard Storage. Call as @data_source.

Parameters:

Name Type Description Default
_func Optional[Func]

Function to be decorated.

 None
storage_type Optional[StorageType]

Storage type to use. This will override the storage type parameter, defined in dashboard instance.

 None

Where data is stored#

When using the @data_source decorator, the function is processed and the data is stored in the Platform Storage, this allows our Front End to retrieve the data quickly and efficiently.

Examples#

You can use the DataSource decorator in two ways:

  • Directly, where the function returns a DataFrame to be used by the widget, without any filter.
  • Indirectly, where the function returns a DataFrame for each option of the widgets that is used as a filter.

Basic example#

Let's recall how one Widget can display data adding the DataFrame directly to the widget. This is the easiest way to view your data.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
import pandas as pd
from engineai.sdk.dashboard.dashboard import dashboard
from engineai.sdk.dashboard.widgets import table

data = pd.DataFrame(
    {
        "country": ["China", "United States", "India"],
        "population": [1416710000, 331915076, 1366418000],
        "gdp_per_capita_usd": [10108, 68318, 2247],
    },
)

dashboard.Dashboard(
    app_slug='engineai-docs-steps', workspace_slug='engineai-docs-steps',
    content=table.Table(data)
)

Let's now expand on this and see how to use the DataSource decorator to get the data from a database and filter it using one or multiple widgets.

Examples use a Postgres Database as data source.

Let's connect to a hypothetical postgres database, where there is a table info with the following information:

country population gdp_per_capita_usd continent top_sector
China 1,416,710,000 10,108 Asia Manufacturing
United States 331,915,076 68,318 North America Technology & Services
India 1,366,418,000 2,247 Asia Agriculture & Services
Brazil 213,993,437 6,767 South America Agriculture & Manufacturing
Russia 145,912,025 11,743 Europe Energy & Natural Resources
Japan 125,507,472 48,870 Asia Technology & Manufacturing
Germany 83,190,556 50,965 Europe Manufacturing & Exports
United Kingdom 68,207,116 41,486 Europe Finance & Services
South Africa 60,041,994 5,815 Africa Mining & Manufacturing
Australia 25,864,475 52,992 Oceania Mining & Natural Resources

Display data from a Database#

The following example shows how to display data from a database using the @data_source decorator. This is the simplest case, where the function returns a DataFrame to be used by the widget and fetching the data directly from the database.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
import pandas as pd
import psycopg2

from engineai.sdk.dashboard.dashboard import dashboard
from engineai.sdk.dashboard.data import data_source
from engineai.sdk.dashboard.widgets import table

# init connection
conn = psycopg2.connect(**params)


@data_source
def get_database_data():
    return pd.read_sql("SELECT * FROM info", conn)


table_1 = table.Table(
    get_database_data(),
    columns=["country", "population", "gdp_per_capita_usd", "continent", "top_sector"],
)

dashboard.Dashboard(
    app_slug='engineai-docs-steps', workspace_slug='engineai-docs-steps',
    content=table_1
)

As you can see, the function get_database_data() returns a DataFrame with the data from the database, and the Table widget displays the data.

Since we are using @data_source decorator to get the data, we need to specify the columns we want to display because the Table widget will not be able to infer them from the data.

Filter using one Widget#

Now let's see how to filter data using a widget. In this case, we are going to use a Toggle widget to filter the data by continent.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
import pandas as pd
import psycopg2

from engineai.sdk.dashboard.dashboard import dashboard
from engineai.sdk.dashboard.data import data_source
from engineai.sdk.dashboard.widgets import table
from engineai.sdk.dashboard.widgets import toggle

# init connection
conn = psycopg2.connect(**params)


@data_source
def get_database_data(toggle_continent):
    return pd.read_sql(
        f"SELECT * FROM info WHERE continent = '{toggle_continent}'", conn
    )


tggl = toggle.Toggle(
    data=pd.DataFrame(data=[{"id": "Asia"}, {"id": "Europe"}]),
    label="Continent",
)

table_1 = table.Table(
    get_database_data(tggl.selected.id),
    columns=["country", "population", "gdp_per_capita_usd", "continent", "top_sector"],
)

dashboard.Dashboard(
    app_slug='engineai-docs-steps', workspace_slug='engineai-docs-steps',
    content=[tggl, table_1]
)

Since Toggle widget is used as a filter, you need to specify in the @data_source function an argument that will receive the tggl options, and where you call the function you need to pass the tggl.selected.id as a parameter. This way, when the Dashboard is being published, the parameter will assume the tggl options (in this case, Asia or Europe).

How the data is stored

Since the @data_source function is receiving the tggl options, the data will be stored in the Platform Storage for each option. This way, when the user selects an option, the Platform fetches the right option.

For example, if the user selects Asia, the Platform will fetch the data stored in the Platform Storage for the path/to/Asia.json option, and if the user selects Europe, the Platform will fetch the path/to/Europe.json option.

Filter using multiple widgets#

Now let's add another widget to filter the data, in this case you are going to use another Toggle widget to filter the data by gdp per capita.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
import pandas as pd
import psycopg2

from engineai.sdk.dashboard.dashboard import dashboard
from engineai.sdk.dashboard.data import data_source
from engineai.sdk.dashboard.widgets import table
from engineai.sdk.dashboard.widgets import toggle

# init connection
conn = psycopg2.connect(**params)


@data_source
def get_database_data(toggle_continent, toggle_gdp):
    return pd.read_sql(
        f"""
        SELECT * FROM info 
        WHERE continent = '{toggle_continent}' AND gdp_per_capita_usd > '{toggle_gdp}'
        """,
        conn,
    )


tggl = toggle.Toggle(
    data=pd.DataFrame(data=[{"id": "Asia"}, {"id": "Europe"}]),
    label="Continent",
)

tggl_2 = toggle.Toggle(
    data=pd.DataFrame(data=[{"id": "1000"}, {"id": "20000"}]),
    label="GDP Per Capita (USD)",
)

table_1 = table.Table(
    get_database_data(tggl.selected.id, tggl_2.selected.id),
    columns=["country", "population", "gdp_per_capita_usd", "continent", "top_sector"],
)


dashboard.Dashboard(
    app_slug='engineai-docs-steps', workspace_slug='engineai-docs-steps',
    content=[tggl, tggl_2, table_1]
)

Same as before, add the toggle_gdp parameter to the @data_source function and pass the tggl_2.selected.id as a parameter when calling the function. Change the SQL query to filter the data by gdp_per_capita_usd and continent.

When publishing the Dashboard, the Platform will store the data in the Platform Storage for each combination of option of the tggl and tggl_2 widgets.