Need help in creating a connector to Hadoop (Hive)

Need help in creating a connector to Hadoop (Hive)

  

Hi,

I am trying to fetch data from Hadoop. There is no built-in  connector available for Hadoop in Outsystems. I  have  3 options in hand.

1. Ingest data from Hadoop to SQLServer

2. Create a rest API and consume the data

3. Create an extension in  Outsystems for the connector


I have been able to achieve both options #1 & #2 , but these are able to temporarily bridge the gap. The permanent solution would be to create an extension. 


I would like to know if anyone has created something similar, as I am not sure where to start. 

I looked at the Master class for integration and extensibility and have got some idea about how to create and debug extensions but still stuck on how to create a DB connection.


Thanks in advance

Hello Sindhuja,

If you are confused as how to create an interface to external database, I recommend you reading this:

https://success.outsystems.com/Documentation/10/Extensibility_and_Integration/Integrate_with_an_External_Database

Let us know if you were able to connect to the database or if you are still having trouble.

Cheers

Sindhuja,

Let me ask you something. The hadoop (HIVE) is not a common database system, right?
I already worked with Hadoop (in an old project), and this HIVE is like a "wrapper" to it...

So, in order to "connect" to it directly through an extension in OutSystems, you would need to create your own Connector, but I think the sdk to do it is not generally available, and you would need to have depth knowledge on the connections and so on.

If that is the case, you probably will be better staying with the API method.

Cheers.

Thanks Eduardo, I am able to connect to external databases in general. Its just that the connector for Hive is not available in Outsystems. So, yeah the requirement is to  create a new connector. I have some knowledge of JAVA and the DB connectivity using JDBC connectors. I was looking if there was a half cooked connector for any DB ( not Exactly Hadoop) out there so that I can build upon it or customize it to suit my needs.

Solution

Hi Sindhuja,

Thake a look in this topic: https://www.outsystems.com/forums/discussion/14992/the-postgresql-database-connector-by-ardoric/

Ricardo did one for the PostgreSQL.

You will need the Database Integration API.

Cheers.

Solution

Thank you Eduardo

Sindhuja Srinivasan wrote:

Thank you Eduardo

Hi. 

Could you give us a feedback if you succeeded in building this connector?


Hi Wanderson,


The simple answer is NO.

But, below is the summary of my journey towards the  conclusion, which may guide you:

The first problem we were facing was that our stack was in .NET . To create a connector in .NET environment, you need a ADO.NET. Hadoop doesnt have it. But, we found that there were some third party (paid) connectors. We decided to proceed with that.

The next problem was to get access to SDK ( from OutSystems support portal). Our stack is in cloud, and experimenting with a third party driver on it was not a good idea. So, outsystems team suggested we create an on-prem environment, only then would they grant the access.

So, we decided its the end of the road.


Your problem will be easy to solve, if you have a JAVA stack.



Sindhuja Srinivasan wrote:

Hi Wanderson,


The simple answer is NO.

But, below is the summary of my journey towards the  conclusion, which may guide you:

The first problem we were facing was that our stack was in .NET . To create a connector in .NET environment, you need a ADO.NET. Hadoop doesnt have it. But, we found that there were some third party (paid) connectors. We decided to proceed with that.

The next problem was to get access to SDK ( from OutSystems support portal). Our stack is in cloud, and experimenting with a third party driver on it was not a good idea. So, outsystems team suggested we create an on-prem environment, only then would they grant the access.

So, we decided its the end of the road.


Your problem will be easy to solve, if you have a JAVA stack.



My problem will not be easy to solve because we use .NET

Installing a third-party solution would not be the problem until I'm trying to experiment with Airlock to connect to Hive. If I succeed, I'll let you know.