Datadance is a simple data processing server that takes a JSON input, performs simple operations, and returns an output.
Problem that we are trying to solve: In many use cases involving an ETL pipeline, we need to execute third-party/user code. Running it in isolation and maintaining it is an operations and security nightmare. DataDance addresses this issue by developing a unique solution where the third-party/user logic can be provided in the form of DataDance transforms. These transforms are not only easy to maintain but also do not use any programming language shell. Internally, it uses a parser that executes the given transforms on the input JSON and provides an output JSON.
The initial state of the project: We developed a TypeScript package to achieve the data transformations. We researched multiple existing tools trying to solve a similar problem and understood the shortcomings of their approaches to incorporate improvements in our project.
Current stage of the project: We now have a UI playground where people can experience DataDance. We also developed a CLI version, and the binary can be easily integrated into various environments. Additionally, we created an API server with several endpoints to save, process, and run transforms. Using the API, a user can test transforms on a sample JSON, and then tune it accordingly. Once ready, the user can save the transforms and run the transform logic on thousands of records at runtime.
Try out our playground at: https://datadance.app/design/build
Or, send a POST request to our API at: https://datadance.app/api/process
Issues, PRs and Discussions
