I couldn't agree more with you on most of these points. You may be interested in trying out Shipyard (www.shipyardapp.com). Fair disclosure, I'm the co-founder. While we don't address all of these issues, we're building with these key focuses.
- Simplicity is key. Data Teams should focus on creating solutions, not fighting infrastructure and limitations.
- Workflows shouldn't change how code is written. Your code should run the same locally as on our platform, with no extra packages or proprietary setup files required.
- Templates are a first-class object. The modern data pipeline should be built with repeatability in mind.
- Data solutions should be usable and visible beyond the walls of technical teams.
We're in a private beta and rapidly trying to improve the product. I would love to chat more if you're interested. Details in profile.
For your specific problems:
1) We're cloud-native and handle hosting/scaling on our side. You don't have to worry about setup. Just log in and launch your code.
2) Our UI is pretty slick (built in AntD) and built to reduce the overwhelming options when setting jobs up.
3) If you want to run a job on-demand, just press "Run now". If you schedule a job, then change the schedule or status, we'll automatically add/update/remove the schedules. Other technical defaults aren't options right now because we're trying to abstract those choices away so Data Teams can just focus on building solutions that work.
4) We let you group your scripts into "Projects" (essentially folders) with a high level overview of the quantity of jobs, as well as how many recently failed or succeeded.
5) Workflows get made directly in the UI. This makes it easier for any less technical users to set up jobs on their own. We still have a lot of improvement to go in this area though.
6) Our product is written in Go (the language of the cloud). We don't force the worker to be in a language and managed by a process in that language. We manage at the process level.
7) Every job creates a new container on the fly, installing package dependencies that the user specifies. You can connect your scripts together without worrying about conflicting packages, conflicting language versions, or without needing to know how to make and manage Docker containers.
8) Not sure how we compare on the logging front. However, we separate out logs for each time a script runs so you're not having to search for a needle in a haystack. You can filter and search for specific logs in the UI.
How much does Shipyard cost? The website doesn't mention pricing at all and I am wary of spending time on anything like this without having a sense of how much it will cost me down the road.
We're still working on defining the exact pricing, but it would be a usage-based monthly subscription model, since our core costs are related to infrastructure scaling.
The good news is that if you had to switch workflow tools, our product design makes your code easily portable (no proprietary config files, no packages to add to your code). You would just lose access to the scaled workflow and execution logic.
I'd love to discuss more with you to see what your team would see as reasonable. Happy to work something out.
- Simplicity is key. Data Teams should focus on creating solutions, not fighting infrastructure and limitations. - Workflows shouldn't change how code is written. Your code should run the same locally as on our platform, with no extra packages or proprietary setup files required. - Templates are a first-class object. The modern data pipeline should be built with repeatability in mind. - Data solutions should be usable and visible beyond the walls of technical teams.
We're in a private beta and rapidly trying to improve the product. I would love to chat more if you're interested. Details in profile.
For your specific problems:
1) We're cloud-native and handle hosting/scaling on our side. You don't have to worry about setup. Just log in and launch your code.
2) Our UI is pretty slick (built in AntD) and built to reduce the overwhelming options when setting jobs up.
3) If you want to run a job on-demand, just press "Run now". If you schedule a job, then change the schedule or status, we'll automatically add/update/remove the schedules. Other technical defaults aren't options right now because we're trying to abstract those choices away so Data Teams can just focus on building solutions that work.
4) We let you group your scripts into "Projects" (essentially folders) with a high level overview of the quantity of jobs, as well as how many recently failed or succeeded.
5) Workflows get made directly in the UI. This makes it easier for any less technical users to set up jobs on their own. We still have a lot of improvement to go in this area though.
6) Our product is written in Go (the language of the cloud). We don't force the worker to be in a language and managed by a process in that language. We manage at the process level.
7) Every job creates a new container on the fly, installing package dependencies that the user specifies. You can connect your scripts together without worrying about conflicting packages, conflicting language versions, or without needing to know how to make and manage Docker containers.
8) Not sure how we compare on the logging front. However, we separate out logs for each time a script runs so you're not having to search for a needle in a haystack. You can filter and search for specific logs in the UI.