re: ETL framework - you can get a lot done with the built-in Airflow Operators, including the PythonOperator (and bring in any python dependency you like) and BashOperator (call any CLI, etc.) - it's not drag-and-drop, but I've found it to be quite versitile.
re: idempotency - yes, make your workflow tasks idempotent.
re: latency - this is being worked on very actively. Ash (PMC member) has committed to working on task latency almost exclusively until it's resolved
re: idempotency - yes, make your workflow tasks idempotent.
re: latency - this is being worked on very actively. Ash (PMC member) has committed to working on task latency almost exclusively until it's resolved
re: sensors, there is some great work from Airbnb to improve: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17+A...