02d8d8367a
* Add initial dataflow template code + TF infra * Refactor the datapipeline DAG to use flex template operator, cleanup code * Remove unneeded bash scripts, update README with manual examples * Refactor datapipeline_dc_tags.py and include new Flex template * Update docs to reflect changes * Remove sub-dependencies and keep apache beam * Add missing license headers and update tests * Set resouces to 291 in tests * Update outputs via tfdoc * Update with outputs order and tfdoc * Correct number of resources * Fix to add region into command from var * Enable service account impersonation for running builds * Update example dataflow run command to use orchestrator SA * Remove hard coded values in example * Keep original airflow files, add new which use Flex template as example * Update tests and doc * Fix number of resources in plan * Run tfdoc remove files section in README * Fix number of modules in tfdoc * Update number of resources * Add missin service account * Update DF demo README * Quick rename --------- Co-authored-by: lcaggio <lorenzo.caggioni@gmail.com> Co-authored-by: Ludovico Magnocavallo <ludomagno@google.com>
30 lines
1.2 KiB
Docker
30 lines
1.2 KiB
Docker
# Copyright 2023 Google LLC
|
|
#
|
|
# Licensed under the Apache License, Version 2.0 (the "License");
|
|
# you may not use this file except in compliance with the License.
|
|
# You may obtain a copy of the License at
|
|
#
|
|
# https://www.apache.org/licenses/LICENSE-2.0
|
|
#
|
|
# Unless required by applicable law or agreed to in writing, software
|
|
# distributed under the License is distributed on an "AS IS" BASIS,
|
|
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
|
# See the License for the specific language governing permissions and
|
|
# limitations under the License.
|
|
|
|
FROM gcr.io/dataflow-templates-base/python39-template-launcher-base
|
|
|
|
ENV FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE="/template/requirements.txt"
|
|
ENV FLEX_TEMPLATE_PYTHON_PY_FILE="/template/csv2bq.py"
|
|
|
|
COPY ./src/ /template
|
|
|
|
RUN apt-get update \
|
|
&& apt-get install -y libffi-dev git \
|
|
&& rm -rf /var/lib/apt/lists/* \
|
|
&& pip install --no-cache-dir --upgrade pip \
|
|
&& pip install --no-cache-dir -r $FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE \
|
|
&& pip download --no-cache-dir --dest /tmp/dataflow-requirements-cache -r $FLEX_TEMPLATE_PYTHON_REQUIREMENTS_FILE
|
|
|
|
ENV PIP_NO_DEPS=True
|