Exploring unstructured data with Endeca on Twitter


We recently talked about exploring unstructured data with Endeca Information discovery using the example of a football game. As Gitex technology week is taking place this week, we thought it was a good opportunity to look at the actual business benefit of this tool.

1. How many people talked about Gitex on Twitter?
Most tweets could be seen at 11h on Monday, from 13-17h attention stayed constant around 350/h. Please note that you have to add +2h for the time difference in Dubai. This information can measure the impact and success of your event.


2. World map of tweets
While as expected most tweets came from the middle east countries (see second map), we could also see lots of messages from Melbourne/Sidney, New York, Puerto Rico, Germany and Brazil. Endeca will capture a relative number of tweets from each country to analyse positive, negative and neutral tweets about a certain keyword. In our map we just got an overall view of where most Gitex tweets came from, but with an EQL statement you can easily add the evaluation and get concrete tweets. This tool gives you the possibility to get opinions from your visitors.



3. Tag cloud
If we have a closer look at what people were talking about in their tweets, we can detect that the mood was positive at the 33rd edition of the technology week. Even better, we got the opportunity to see which of the 3500 stands stuck out. For example, we can see the terms dubai police, visiting dubai police and black points; the story behind these words is a promotion where every visitor of the Dubai police stand would get 8 black points deducted of their driver´s license for free. The topic went viral and probably caused high traffic at their stand.
From the second day tag cloud we can see that the Abu Dhabi Judicial Department (adjd) was mentioned, during the last years they raised lots of attention because of their keen interest in developing judicial services through more focused investment in IT systems and applications. Later on Todd Carpenter can be found in the cloud, probably because the industry veteran just got rehired for the new predictive analytics department at NAR.

It´s astonishing to see all the details we were able to figure out just by using some of the Endeca functions. We will continue tu use the tool to see what other benefits we can get out of this.

Endeca information discovery

Endeca information discovery

If you want to find out more about how Endeca Information discovery can give you inside information about your brand from social media and blogs, leave us a comment or contact us at info@clearpeaks.com.

How to access the DAC server using the command line


One of the features that DAC has is the ability to remotely access it through the command line. This allows us to get information such as the status of an ETL execution or the status of the Informatica server. Moreover we can also start or stop execution plans and create shell scripts that can call the DAC server and programme actions depending on the status of a job.

In this article we are going to see how to set up the command line access to the DAC server and how to use the commands available. Finally we will see how to create a script that automatically restarts an ETL execution if it has failed.

Setting up the access to the DAC server:

First of all make sure that you have installed in your local machine a supported version of JAVA SDK.

Next go to the DAC server and navigate to the DAC folder (C:\OracleBI\DAC) and copy the following files in a new folder in your local machine:

  • DAWSystem.jar: DAC executable Jar file
  • dac.properties: File that contains the connection properties to access the DAC server
  • dacCmdLine.bat: Batch file that contains the connection string to the DAC server

Edit the dac.properties file with your DAC server connection properties (syntax code for windows environment):




You can find the RepositorySampVal in the DAC console by clicking on Help – Login details

DAC server access

Now edit the dacCmdLine.bat and set the JAVA_HOME and DAC_HOME (folder where you placed the DAC files from the server) path of your local machine.

set JAVA_HOME=C:\Java\jdk1.5.0_17

set DAC_HOME=C:\ DACRemote

set JAVA=%JAVA_HOME%\bin\java.exe

Make sure that your paths do not contain any space, as it would fail during execution.

Now we are ready to access the DAC server from our local machine.

Using the command line to access to the DAC server:

Open a new command line window and from your local DAC folder execute the following command:

-> dacCmdLine EtlStatus

DAC server access

The command will return a summary of the last subject area executed in the server.

There are five commands that we can use to interact with the DAC server:

  • EtlStatus: Returns a summary of the last subject area executed in the server, also you can specify which subject area you want to see the status by adding the name of the subject area as a second parameter.
  • StartEtl: Starts an execution plan. You have to specify the subject area as the second parameter.
  • StopEtl: Stops an execution plan. You have to specify the subject area as the second parameter.
  • Databasestatus: Checks if the DAC server can connect to all the databases.

DAC server access

  • InformaticaStatus: Checks if the DAC Server can connect to the Informatica server.

Acces DAC server

Building a script to automatically restart a failed ETL execution:

Sometimes due to errors in the network the ETL execution fails as the communications between the source or target DB and the DAC server is lost. In these cases when there is nothing wrong in our mappings or DAC tasks we only need to restart the ETL to complete the execution successfully.

To avoid restarting manually the ETL each time it fails we will schedule a task that checks the status of the ETL execution and restarts it if it has failed.

Create a new batch file and name it ‘Restart_Failed_ETL.bat’, make sure that you place the file in a server where you can schedule batch tasks and also that has the remote DAC installation done (in case it is not the ETL server itself).

Open the batch file and first set the paths to JAVA and DAC:

set JAVA_HOME=C:\Java\jdk1.5.0_17

set DAC_HOME=C:\ DACRemote

set JAVA=%JAVA_HOME%\bin\java.exe

Additionally create a new variable to set the name of the subject area we want to start:

set ETL_LOAD="CRM – Daily Load"

The next line will get the ETL status and save it in the variable RESULT. Notice that we look for the string message as it is after that when we get the status of the ETL.

for /f "tokens=2" %%i in ('%JAVA% -cp %DAC_HOME%\DAWSystem.jar com.siebel.etl.net.DACCommandLine %DAC_HOME%\dac.properties ETLStatus ^| findstr Message:') do SET RESULT=%%i 

Now we will check if the status message contains the word ‘some’ as it is the first word of the message that the ETL returns if it has failed:

Message: Some steps failed.

If it contains the word ‘some’ then it restarts the ETL again by executing the StartETL command:

IF %RESULT%==Some (%JAVA% -cp %DAC_HOME%\DAWSystem.jar com.siebel.etl.net.DACCommandLine %DAC_HOME%\dac.properties StartETL %ETL_LOAD%)

ELSE ( ECHO ETL will not be restarted)

The final script should look like this:

ECHO off

set JAVA_HOME=E:\Java\jdk1.5.0_17

set DAC_HOME=E:\OracleBI\DAC

set JAVA=%JAVA_HOME%\bin\java.exe


for /f "tokens=2" %%i in ('%JAVA% -cp %DAC_HOME%\DAWSystem.jar com.siebel.etl.net.DACCommandLine %DAC_HOME%\dac.properties ETLStatus ^| findstr Message:') do SET RESULT=%%i

IF  %RESULT%==Some (%JAVA% -cp %DAC_HOME%\DAWSystem.jar com.siebel.etl.net.DACCommandLine %DAC_HOME%\dac.properties StartETL %ETL_LOAD%) ELSE ( ECHO ETL will not be restarted)

Finally schedule the execution of the batch file to start at the same time that the ETL normally finishes. Now it will check every night if the ETL has completed or failed and then automatically restart it if it is the case.

We hope you find this information useful, please leave a comment if you have any questions or thoughts.

ClearPeaks event at the chamber of commerce Barcelona


Yesterday ClearPeaks presented the visual analytics tool Tableau to over 50 professionals in the center of Barcelona. We chose the Cambra de Comerç, a modern business center which provided the ideal venue for our networking morning.

We were happy to welcome companies from e-commerce, retail, consulting, entertainment, editorial, real estate and other businesses to tell them about our service and the advantages of visual analytics. Our goal was to present a more convenient solution for creating visual dashboards.

tableau event Clearpeaks

Tableau is a software which is easy to use and allows the user to be more self-sufficient (from IT) than usually. The price for licenses is quite attractive and it´s very suitable for middle sized and small companies.

We brought the presentation to life with a demo of a Spanish e-commerce site, showing sales and traffic on the site per region and hour. It was astonishing how easy it was to create new dashboards with different filters to allow a detailed analysis of marketing campaigns.

Afterwards we enjoyed a networking coffee discussing about technical details and software integration. Questions raised during the event can be found on Twitter with #clearpeaksccb. Join us next time or give us a call for further information about visual analytics.

tableau demo clearpeaks

Español: Evento ClearPeaks en la Camara de Comercio de Barcelona

ClearPeaks presentó la herramienta de analítica visual Tableau a más de 50 profesionales en el centro de Barcelona. Elegimos la cámara de comercio, un moderno centro de negocios ideal para nuestra mañana de networking.

Estamos muy contentos de haber dado la bienvenida a empresas de sectores como e-commerce, retail, consultoría, cultura, editorial, inmobiliaria, entre otros; y haber tenido el placer de presentar las ventajas de las herramientas de analítica visual y los servicios que ponemos a disposición para el buen uso de estas. Nuestro objetivo fue el de presentar una solución que creemos de las más convenientes e innovadoras para la creación de cuadros de mando visuales (dashboards).

Tableau es un software muy fácil de usar, permite al usuario ser más auto-suficiente (independiente de IT) de lo habitual. El precio de las licencias es muy atractivo, siendo además muy adecuado para PYMES.

Con el ejemplo de una empresa de comercio electrónico demostramos las posibilidades de la herramienta. Mediante distintos gráficos pudimos visualizar las ventas y el tráfico en el sitio por región y hora. Fue asombroso ver lo fácil que era crear nuevos cuadros de mando con diferentes filtros para poder realizar un análisis detallado de las campañas de marketing.

Al final de la mañana disfrutamos de un café, discutiendo sobre los detalles técnicos y la integración de software. Las preguntas que surgieron durante el evento se pueden encontrar en Twitter en #clearpeaksccb. 

 Te invitamos a asistir la próxima vez, o llámenos para obtener más información sobre las herramientas de Visual Analytics.

privacy policy - Copyright © 2000-2010 ClearPeaks