AWS – Boris blog!

DBT 1.0.0 on AWS MWAA

user — Sun, 16 Oct 2022 14:25:01 +0000

Initially I thought that it sounds too good to be true: https://docs.aws.amazon.com/mwaa/latest/userguide/samples-dbt.html

Then I decided to give it a try – only to realise that dbt-core=0.21.1 ?!

Then I went for dbt-core=1.3.0 – it didn’t work -> collision with apache-ariflow=2.2.2

Then I tried dbt-core=1.0.0 – that one worked. It will work for you too, just use the following requirements.txt

apache-airflow[postgres,mysql,google]==2.2.2

json-rpc==1.13.0
minimal-snowplow-tracker==0.0.2
packaging==20.9
networkx==2.6.3 
mashumaro==2.9
sqlparse==0.4.2

logbook==1.5.3
agate==1.6.1
dbt-extractor==0.4.0

pyparsing==2.4.7 
msgpack==1.0.4
parsedatetime==2.6
pytimeparse==1.1.8
leather==0.3.4
pyyaml==6.0

# Airflow constraints are jsonschema==3.2.0
jsonschema==3.1.1
hologram==0.0.14
dbt-core==1.0.0

psycopg2-binary==2.8.6
dbt-postgres==1.0.0
dbt-redshift==1.0.0

Atlantis for Terraform with support for automated building of python AWS Lambda Functions [Dockerfile]

user — Wed, 17 Feb 2021 07:42:23 +0000

If you look for a way to use Atlantis both to deploy your infrastructure as a code and package and build AWS Lambda Functions with Python I provide you a Dockerfile for that purpose. Some use cases require, that the Terraform repository holds the infra as code and also the actual source code that this infra should execute. In the case of the Lambda functions, that is both the description for a function (how much memory, timeout, IAM role) as well as the actual code that has to be executed. On the other hand python lambda functions has to be built and zipped locally, especially if they contain external libraries, before uploading them to the cloud. That means, the machine, or cluster that runs Atlantis needs Python to perform the task, and that is what the following image gives you: it pulls the original Atlantis image from Docker Hub and on top of that installs Python.

Important: You probably want to include this (or similar) terraform module to build the python packages: https://github.com/claranet/terraform-aws-lambda

Dockerfile:

FROM runatlantis/atlantis:latest
ENV USER atlantis_deploy
RUN adduser -D $USER
ENV HOME /home/$USER
ENV PY_VERSION='3.7.2'
RUN apk add --no-cache --update \
    curl git bash bash-doc bash-completion \
    openssh libffi-dev \
    openssl-dev \
    bzip2-dev zlib-dev zip \
    readline-dev sqlite-dev \
    build-base \
    linux-headers \
    util-linux \
    docker
WORKDIR /home/$USER
# Install Python 3.7 via pyenv
ARG PYENV_HOME=$HOME/.pyenv
RUN git clone --depth 1 https://github.com/pyenv/pyenv.git $PYENV_HOME && \
    rm -rfv $PYENV_HOME/.git
ENV PATH $PYENV_HOME/shims:$PYENV_HOME/bin:$PATH
RUN pyenv install $PY_VERSION
RUN pyenv global $PY_VERSION
RUN pip install --upgrade pip && pyenv rehash
RUN rm -rf ~/.cache/pip
# Install additional software
RUN pip3 install --upgrade awscli pipenv boto3 click ssm-cache

DIY: AI-Powered Smart Scale Part 1

user — Sat, 02 Jan 2021 19:05:41 +0000

In January, 2020 looked like we (our startup) can optimize the food sector even more with an automated food waste tracking solution. I had already seen some solutions for smart scales and I though: why don’t we build our own, it should not be that hard? And indeed it isn’t, so I would like to tell you everything around it. I hope that you find the information interesting and useful.

Some usecases

Digitalisation of the stock: Keep track of the stock in the kitchen and know exactly how much and what food ingredients are available by measuring the weight of the newly bought products and classifying them via image recognition / QR codes scans.

Expiration date: Track the quality of the food by not only capturing its weight and pictures of it, but also storing its expiration date.

Evaluation of waste: This is a use case for a smart waste bin: the waste bin will be placed on the scale and every time ingredients are thrown away one can track what exactly and how much.

How it works

1. Food waste is disposed as usual in a bin and its weight is measured by the smart scale.

2a. Semi automated: The operator chooses type of food and reason for throwing food away on a touchscreen.

2b. Automated: The scale automatically recognizes the food. The operator can enter the reason for food thrown away and correct the recognition.

3. Data (images, recognitions and measurements) are surfaced in reports, answering questions like WHAT, HOW MUCH and WHY food is thrown away.

Hardware

Before getting the mentioned components I started drawing… As bad drawer as I am, you can get a rough idea where all is heading:

If you want to build the same scale you will need:

Scale Bimco В30/К60 with RS-232 port	http://www.bimco.net/en/Platform-Scales/9/view/
Raspberry Pi 4 Computer Modell B, 2GB RAM (a complete overkill for the scope of the project)
USB (A) to RS-232
Raspberry Pi 4 Screen Case
Raspberry Pi 7-Inch Touch Screen Display
Logitech C270 Webcam
Stainless steel merging part	Custom made

Funny enougth, the stainless steel merging part was hardest for me to get, luckily my neighbour is a craftsman so I was able to get it exactly as needed.

Once I had all the parts, I could finally get my hands dirty. I installed Ubuntu on the computer and once I was able to SHH with Putty into the machine I installed NodeJS to build and run the software. There will be an application on the Raspberry, that will collect the data from the scale, add timestamps, get pictures from the webcamera and push all that data to the AWS cloud. More on that in the next post, where I show you what software I wrote, how I ran it and how I integrated the AI part.

Before merging the Raspberry with the display I still used allmost all ports of the Raspberry. One for powering with USB-C, one USB for the web camera, one for the RS232 to USB cable that streamed the data from the scale and an Ethernet port connecting the Raspberry to my router:

Soon my next blog post for the software will follow. The link to the post will appear here.