Thursday, May 25, 2017

Steps taken to get the Docker Web of Needs Swarm Running

**Actually, it does not seem to be a swam. it is running on one machine, but docker-compose is being used.

Beginning with https://github.com/researchstudio-sat/webofneeds/tree/master/webofneeds/won-docker I followed the steps:

(1) Install docker
Start with https://docs.docker.com/get-started/

(2) Install docker-compose
https://docs.docker.com/compose/overview/
and follow the tutorial
https://docs.docker.com/compose/gettingstarted/

(3) Create base_folder (mkdir data), .env file (see below), and docker-compose file (wget https://raw.githubusercontent.com/researchstudio-sat/webofneeds/master/webofneeds/won-docker/deploy/local/docker-compose.yml)

~/Documents/dockerexperiments/webofneeds$ ls -a
.  ..  data  docker-compose.yml  .env

~/Documents/dockerexperiments/webofneeds$ cat .env

deploy_host=localhost
base_folder=./data/



(4) Run docker compose in the same directory without the detached option to enable debugging:
sudo docker-compose up (also "sudo service tomcat7 stop" and "sudo service mongodb stop" to prevent port conflicts)

(5) There is an error when trying to create a resource over https on localhost. Here are some instructions that may help set it up:
https://docs.docker.com/engine/security/https/

(6) Investigate the server cert:
~/Documents/dockerexperiments/webofneeds/data/won-server-certs$ openssl x509 -noout -text -in t-cert.pem
Certificate:
    Data:
        Version: 3 (0x2)
        Serial Number: 9744456803442286009 (0x873b43d59a50ddb9)
    Signature Algorithm: sha256WithRSAEncryption
        Issuer: CN=localhost
        Validity
            Not Before: May 25 00:12:13 2017 GMT
            Not After : May 25 00:12:14 2018 GMT
        Subject: CN=localhost
        Subject Public Key Info:
            Public Key Algorithm: rsaEncryption
                Public-Key: (2048 bit)
                Modulus:
                    00:f5:a7:56:1a:c6:91:dc:3e:ed:82:ec:bc:73:87:
                    89:34:fc:e7:63:d8:e3:b6:21:0b:76:db:44:f8:6d:
                    5c:e7:e1:64:2b:48:76:a1:2e:b2:fe:28:dd:8b:3f:
                    8c:4c:1d:42:ca:bf:92:c0:90:fe:05:a6:db:2a:d6:
                    17:86:dd:5d:76:a0:86:7f:c2:ec:32:85:0a:91:d3:
                    7d:52:bf:43:bf:c1:84:8b:e0:30:e6:08:39:1c:c2:
                    63:3c:49:c2:11:8b:2e:b9:f9:39:78:40:59:4c:a2:
                    d7:b5:f4:c8:5b:68:72:72:51:f1:2d:8f:e7:0d:f5:
                    de:f7:38:42:4a:13:b0:23:c4:58:61:bc:1a:c0:b0:
                    09:38:4b:aa:06:76:b2:67:24:80:b2:71:81:bd:5e:
                    55:e4:6c:86:14:62:71:e5:e0:c8:97:f2:59:17:41:
                    b1:78:44:49:8c:28:bb:ab:9c:72:e1:7b:ec:7c:9b:
                    b5:31:da:66:18:8e:9e:61:89:a8:29:79:84:89:a2:
                    4f:dc:c0:39:4e:a8:e3:35:e1:0e:b8:2f:26:c8:2a:
                    7a:ad:11:23:25:5b:39:cc:81:bd:4b:66:b7:aa:31:
                    1c:69:79:6f:e6:03:fe:c0:eb:c9:98:6a:a6:32:3c:
                    b8:3e:c3:2b:f4:6b:34:66:6c:ae:ba:8f:b0:ef:23:
                    c0:b1
                Exponent: 65537 (0x10001)
        X509v3 extensions:
            X509v3 Subject Key Identifier:
                36:B7:33:A7:6D:AC:CE:1C:99:51:84:20:64:F4:A2:62:98:97:32:71
            X509v3 Authority Key Identifier:
                keyid:36:B7:33:A7:6D:AC:CE:1C:99:51:84:20:64:F4:A2:62:98:97:32:71

            X509v3 Basic Constraints:
                CA:TRUE
    Signature Algorithm: sha256WithRSAEncryption
         be:73:e7:58:dd:78:38:36:56:14:90:bf:da:f1:ca:1f:1e:32:
         bf:92:8f:e7:ee:16:48:38:13:83:8d:ce:d2:ec:fd:45:9b:31:
         eb:b2:07:d0:b6:bf:5c:78:32:89:a2:e1:0c:18:ce:d0:19:ee:
         ae:75:85:6f:da:a5:81:d6:f2:aa:01:b1:65:33:ce:9d:ec:99:
         9f:3a:69:3f:66:68:c1:39:27:5f:23:ae:26:9b:60:d7:ef:48:
         74:41:b0:d2:0c:c9:fc:2c:7a:70:35:75:70:6f:71:64:72:78:
         d9:b1:b9:fb:f3:4e:45:a0:5c:83:b0:c4:35:25:7a:41:b3:4a:
         21:34:67:2d:af:28:1f:66:64:42:ab:e5:46:30:4a:6f:8e:d9:
         5a:62:bf:04:d5:fb:e7:3a:ca:86:60:35:9e:93:b3:94:89:81:
         d0:6a:5f:c9:80:27:c2:b2:88:54:a8:61:c9:b6:47:07:c0:3b:
         91:26:8e:7f:c7:62:44:48:a9:5b:30:96:27:22:1f:9c:ad:dd:
         30:42:b2:a7:71:39:2b:30:85:7b:0f:f4:8a:36:1a:ec:75:96:
         6e:53:c4:2e:f5:28:2c:57:de:e8:56:82:84:99:e1:1b:f0:d3:
         1d:6b:37:d6:53:da:e6:74:00:bb:69:f8:64:ba:83:b8:5e:49:
         7f:91:f4:60

========================
 other interesting things
======================================

Interesting part of the log:
owner_1            | 25-May-2017 19:35:01.759 INFO [localhost-startStop-1] org.apache.jasper.servlet.TldScanner.scanJars At least one JAR was scanned for TLDs yet contained no TLDs. Enable debug logging for this logger for a complete list of JARs that were scanned but no TLDs were found in them. Skipping unneeded JARs during scanning can improve startup time and JSP compilation time.

Interesting part of the log:
java.util.concurrent.TimeoutException

Interesting part of the log:
Connection refused:

Interesting part of the log:
blue hard drive light on...perhaps the RAM is low and SWAP space is being used, perhaps there are memory leaks

Fix the certificate due to https://localhost:8890 unable to create resource errors?:

/home/brent/Documents/dockerexperiments/webofneeds/data/won-server-certs# cp *.pem /home/brent/.docker/

(will this really work? The "Protect the Docker daemon socket" has a secure by default section that says to set the DOCKER_HOST and DOCKER_TLS_VERIFY variables (setting DOCKER_TLS_VERIFY=1 in the .env file causes an error))

Friday, May 19, 2017

PowerAqua test May 19th

<?xml version="1.0" encoding="UTF-8"?>
<CONFIGURATION>
        <WATSON>false</WATSON>
        <POWERMAP>false</POWERMAP>
        <VIRTUOSO>false</VIRTUOSO>
        <REMOTESPARQLVIRTUOSO>false</REMOTESPARQLVIRTUOSO>
<ONTOLOGY_INDEX_DB>jdbc:mysql://127.0.0.1:3306/ontologyindexrelation</ONTOLOGY_INDEX_DB>
<ONTOLOGY_INDEX_DB_LOGIN>root</ONTOLOGY_INDEX_DB_LOGIN>
<ONTOLOGY_INDEX_DB_PASSWORD>password</ONTOLOGY_INDEX_DB_PASSWORD>
<INDEX_GLOBAL_PATH>./LuceneIndexes/</INDEX_GLOBAL_PATH>
<INDEX_INFO_FOLDER>./indexListInformation/indexbrentweb/</INDEX_INFO_FOLDER>


running: http://localhost:8080/poweraqua gives :

no error...


Clicking the ask button (with a string in the blank) gives:





java.lang.NullPointerException
 poweraqua.LinguisticComponent.GateChunker.ExecuteSerialController(GateChunker.java:322)
 poweraqua.LinguisticComponent.GateChunker.ParseQuestion(GateChunker.java:445)
 poweraqua.LinguisticComponent.LinguisticComponent.parseQuestion(LinguisticComponent.java:114)
 org.apache.jsp.jsp.answer_jsp._jspService(answer_jsp.java:167)
 org.apache.jasper.runtime.HttpJspBase.service(HttpJspBase.java:70)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:727)
 org.apache.jasper.servlet.JspServletWrapper.service(JspServletWrapper.java:432)
 org.apache.jasper.servlet.JspServlet.serviceJspFile(JspServlet.java:395)
 org.apache.jasper.servlet.JspServlet.service(JspServlet.java:339)
 javax.servlet.http.HttpServlet.service(HttpServlet.java:727)

Monday, May 15, 2017

Change Log for PowerAqua (may 15th)

In /var/lib/tomcat7/webapps/poweraqua, change service_properties.xml from:

<?xml version="1.0" encoding="UTF-8"?><CONFIGURATION><PLUGIN_MANAGER>/var/lib/tomcat7/webapps/poweraqua/WEB-INF/aquaplugins</PLUGIN_MANAGER><REPOSITORY><SERVER>127.0.0.1:8080/openrdf-sesame</SERVER><PROXY></PROXY><PORT></PORT><LOGIN></LOGIN><PASSWORD></PASSWORD><PLUGIN_TYPE>sesame2</PLUGIN_TYPE><REPOSITORY_NAME>isp</REPOSITORY_NAME><TYPE>OWL</TYPE></REPOSITORY></CONFIGURATION>

to:

<?xml version="1.0" encoding="UTF-8"?><CONFIGURATION><PLUGIN_MANAGER>/var/lib/tomcat7/webapps/poweraqua/WEB-INF/aquaplugins</PLUGIN_MANAGER><REPOSITORY><SERVER>http://127.0.0.1:8080/openrdf-sesame</SERVER><PROXY></PROXY><PORT></PORT><LOGIN></LOGIN><PASSWORD></PASSWORD><PLUGIN_TYPE>sesame2</PLUGIN_TYPE><REPOSITORY_NAME>isp</REPOSITORY_NAME><TYPE>OWL</TYPE></REPOSITORY></CONFIGURATION>

(note, I might need to use localhost instead of 127.0.0.1)

Also in /var/lib/tomcat7/webapps/poweraqua/indexListInformation/indexbrentweb, change service_properties.xml from:

<?xml version="1.0" encoding="UTF-8"?>
<CONFIGURATION>
<PLUGIN_MANAGER>/var/lib/tomcat7/webapps/poweraqua/WEB-INF/aquaplugins</PLUGIN_MANAGER>
<REPOSITORY>
    <SERVER>127.0.0.1:8080/openrdf-sesame</SERVER>
    <PROXY></PROXY>
    <PORT></PORT>
    <LOGIN></LOGIN>
    <PASSWORD></PASSWORD>
    <PLUGIN_TYPE>sesame2</PLUGIN_TYPE>
    <REPOSITORY_NAME>isp</REPOSITORY_NAME>
    <TYPE>OWL</TYPE>
</REPOSITORY>
</CONFIGURATION>

to:

<?xml version="1.0" encoding="UTF-8"?>
<CONFIGURATION>
<PLUGIN_MANAGER>/var/lib/tomcat7/webapps/poweraqua/WEB-INF/aquaplugins</PLUGIN_MANAGER>
<REPOSITORY>
    <SERVER>http://127.0.0.1:8080/openrdf-sesame</SERVER>
    <PROXY></PROXY>
    <PORT></PORT>
    <LOGIN></LOGIN>
    <PASSWORD></PASSWORD>
    <PLUGIN_TYPE>sesame2</PLUGIN_TYPE>
    <REPOSITORY_NAME>isp</REPOSITORY_NAME>
    <TYPE>OWL</TYPE>
</REPOSITORY>
</CONFIGURATION>

After running the shell script: /var/lib/tomcat7/webapps/poweraqua# ./createLuceneIndex.sh

the following log results:
https://gist.github.com/bshambaugh/d00017026e916d917030628c0ebec7b7

tl;dr: it helped a lot to put http in the server path before 127.0.0.1, so I avoid line 1152 "java.lang.IllegalStateException: unsupported protocol: '127.0.0.1'" in  https://gist.github.com/bshambaugh/c06f3b2b784f47e5a6e123df2ab42fed

./createLuceneIndex.sh also produced the following changes to the MySQL table:


But ontologyindextable is empty :( .



Backup of the ontology on gist: https://gist.github.com/bshambaugh/04581f98faa875e87c96684566de7ad8

Loading PowerAqua at http://localhost:8080/poweraqua/jsp/init.jsp gives:




What I worked on Yesterday

I played with jQuery dialog boxes yesterday [1], thought about media queries to adjust the CSS grid layout, and read about scaling SVG documents [2] as well as button creation [3] so I can make my own scaleable [2] buttons.

 [1] https://jqueryui.com/dialog/#modal-form
 [2] https://css-tricks.com/scale-svg/
 [3]
https://www.youtube.com/watch?v=iOWzFqrdOVk

Friday, April 28, 2017

Status Log: Still Unable Populate MySQL tables with Lucene Index Script

I'm sharing what I have so far for the createLuceneIndex.sh file. Maybe I'll figure it out eventually.  The error log is linked to at the end.

I created a shell script to create the LuceneIndex at
https://gist.github.com/bshambaugh/42c1accf6d2e5dc607a8d46047cc98be .

I manually created databases ontologyindexrelation and metadataindex.

The script created the following tables in the database metadataindex:

indexbrentwebdirectclasses
indexbrentwebdirectsubclasses
indexbrentwebdirectsuperclasses
indexbrentwebequivalent
indexbrentwebsubclasses
indexbrentwebsuperclasses

The database tables remain empty.
Expect the database tables to contain the columns: id, ontologyId, entityURI, classURI, and classLabel

The database ontologyindexrelation contains no tables.
Expect the table ontologyindextable with the columns: id, ontologyId, and indexManagerId

I manually created ./poweraqua/indexListInformation/indexbrentweb:

with service_properties.xml

<?xml version="1.0" encoding="UTF-8"?>
<CONFIGURATION>
<PLUGIN_MANAGER>/var/lib/tomcat7/webapps/poweraqua/WEB-INF/aquaplugins</PLUGIN_MANAGER>
<REPOSITORY>
    <SERVER>127.0.0.1:8080/openrdf-sesame</SERVER>
    <PROXY></PROXY>
    <PORT></PORT>
    <LOGIN></LOGIN>
    <PASSWORD></PASSWORD>
    <PLUGIN_TYPE>sesame2</PLUGIN_TYPE>
    <REPOSITORY_NAME>isp</REPOSITORY_NAME>
    <TYPE>OWL</TYPE>
</REPOSITORY>
</CONFIGURATION>

and index_properties.xml

<?xml version="1.0" encoding="UTF-8"?>
<CONFIGURATION>
<INDEX>
<INDEX_DIRECTORY>indexbrentweb/index_dir/</INDEX_DIRECTORY>
<SPELL_INDEX_DIRECTORY>indexbrentweb/spell_index_dir/</SPELL_INDEX_DIRECTORY>
<METADATA_INDEX_DB>jdbc:mysql://127.0.0.1:3306/metadataindex</METADATA_INDEX_DB>
<METADATA_INDEX_DB_LOGIN>root</METADATA_INDEX_DB_LOGIN>
<METADATA_INDEX_DB_PASSWORD>your_password</METADATA_INDEX_DB_PASSWORD>
<METADATA_INDEX_TABLE>indexbrentweb</METADATA_INDEX_TABLE>
</INDEX>
</CONFIGURATION>

I copied the index_properties.xml and service_properties.xml to the global files in

./poweraqua/

My multi_index_properties.xml looks like:

<?xml version="1.0" encoding="UTF-8"?>
<CONFIGURATION>
        <WATSON>false</WATSON>
        <POWERMAP>true</POWERMAP>
        <VIRTUOSO>false</VIRTUOSO>
        <REMOTESPARQLVIRTUOSO>false</REMOTESPARQLVIRTUOSO>
<ONTOLOGY_INDEX_DB>jdbc:mysql://127.0.0.1:3306/ontologyindexrelation</ONTOLOGY_INDEX_DB>
<ONTOLOGY_INDEX_DB_LOGIN>root</ONTOLOGY_INDEX_DB_LOGIN>
<ONTOLOGY_INDEX_DB_PASSWORD>your_password</ONTOLOGY_INDEX_DB_PASSWORD>
<INDEX_GLOBAL_PATH>./LuceneIndexes/</INDEX_GLOBAL_PATH>
<INDEX_INFO_FOLDER>./indexListInformation/indexbrentweb/</INDEX_INFO_FOLDER>
</CONFIGURATION>

./LuceneIndexes now contains:

.
└── indexbrentweb
    ├── index_dir
    │   ├── _instances
    │   │   ├── segments_c
    │   │   └── segments.gen
    │   ├── segments_c
    │   └── segments.gen
    └── spell_index_dir
        ├── _instances
        │   ├── segments_1
        │   └── segments.gen
        ├── segments_1
        └── segments.gen

My error log looks like:

https://gist.github.com/bshambaugh/c06f3b2b784f47e5a6e123df2ab42fed
Conclusions:

At this point, I think that commons-http-client-3.1 might be out of date.

I'm ignoring SLF4J errors for now. I ran createSesame2Repository.sh and it successfully created a full repository in sesame (I had to create the repository isp though and change the below part of Sesame2URLDatabaseTransformer.java and recompile to Sesame2URLDatabaseTransformer.class via Netbeans

 public static void main(String[] args)
  {
    String sesameServerPath = "http://kmi-web03:8080/openrdf-sesame/";
    String repositoryName = "evoont-seals_10000K_final";

    String path = "http://kmi.open.ac.uk/technologies/poweraqua/evoont-seals_10000K_final.owl";
    String baseURI = "";
    createDBFromURI(sesameServerPath, repositoryName, path, "RDF", baseURI);
  }

)
This looks weird: Getting Services from 127.0.0.1:8080/openrdf-sesame isp
I changed all localhost instances in the config files to 127.0.0.1 .

Sidenote: I have no idea why this is hardcoded:
"
public static void main(String[] args)
{
RepositoryVirtuoso repository = new RepositoryVirtuoso();
repository.setRepositoryName("jdbc:virtuoso://kmi-dev02.open.ac.uk:8890#http://geography.org");
System.out.print(repository.getGraphIRI());
}"

https://github.com/bshambaugh/PowerAqua-decompiled/blob/master/lib/poweraqua/serviceConfig/RepositoryVirtuoso.java
I did include the downloaded class file in a jar file in the CLASSPATH of createLuceneIndex.sh

/var/lib/tomcat7/webapps/poweraqua/WEB-INF/lib/serviceConfig.jar

Thursday, April 27, 2017

Create a createLuceneIndex.sh file that works: [pt4}:

Remove .DS_Store file:

Remove from the filesystem and the CLASSPATH:
log4j-1.2.15.jar  logback-classic-0.9.9.jar  slf4j-jdk14-1.5.0.jar


I don't know what apache axis is in the log file:

https://gist.github.com/bshambaugh/06abc750eed9346a1f52674bf89c98a0

You can download the used version here?

http://www.apache.org/dyn/closer.cgi/ws/axis/1_4

Try removing the requirement in the classpath and see if the error goes away..

/var/lib/tomcat7/webapps/poweraqua/myLuceneLib/lib/axis-ant.jar  
 /var/lib/tomcat7/webapps/poweraqua/WEB-INF/lib/axis.jar



I get as the response:

https://gist.github.com/bshambaugh/2abbcd0f99457d90106b8ac5535afa80

SO apparently WatsonPlugin.WatsonPlugin  needs axis, which I removed from the CLASSPATH
i.e.: "Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/axis/client/Service ..."

Maybe it is this::



"

Internationalization

If you make changes to the source code that results in the generation of text (error messages or debug information), you must follow the following guidelines to ensure that your text is properly translated.

Developer Guidelines

  1. Your text string should be added as a property to the resource.properties file (xml-axis/java/src/org/apache/axis/i18n/resource.properties). Note that some of the utility applications (i.e. tcpmon) have their own resource property files (tcpmon.properties)." (http://axis.apache.org/axis/java/developers-guide.html) 
Well it looks like /home/brent/Downloads/axis/org/apache/axis/i18n/resource_ja.properties and  /home/brent/Downloads/axis/org/apache/axis/i18n/resource.properties were never changed...

maybe I just need to add the properties file to the class path as described here:
http://stackoverflow.com/questions/2083159/cant-find-bundle-for-base-name

Wednesday, April 26, 2017

Create a createLuceneIndex.sh file that works: [pt3}:


createLuceneIndex.sh now creates:
https://gist.github.com/bshambaugh/b3782d15b24668d8db6531e32d9ee7be

so change in multi_index_properties.xml:

<ONTOLOGY_INDEX_DB>jdbc:mysql://localhost:3306/ontologyindexrelation</ONTOLOGY_INDEX_DB>

<ONTOLOGY_INDEX_DB>jdbc:mysql://127.0.0.1:3306/ontologyindexrelation</ONTOLOGY_INDEX_DB>

and I get:

https://gist.github.com/bshambaugh/56f04274201e00716a8cc4fd0af7fce7

it could be the http-client I am using ( a google search gives me):

https://jira.duraspace.org/browse/DS-1937

https://issues.apache.org/jira/browse/HTTPCLIENT-897

So maybe I need to try a later version of the httpclient???

http://hc.apache.org/

https://mvnrepository.com/artifact/commons-httpclient/commons-httpclient

now this: so change code where appropriate:

https://mvnrepository.com/artifact/org.apache.httpcomponents/httpclient