Saturday, December 28, 2013

Mount Nexus5 to Ubuntu 12.04

Instruction how to connect Nexus phone to Ubuntu 12.04:

1) Activate developer options on Nexus 5, instructions, that mode have to be switched ON always to be mounted.

2) just follow http://bernaerts.dyndns.org/linux/74-ubuntu/268-ubuntu-automount-any-mtp-device , at the end of article there is a script that do all that steps automatically (that require only 2 chapter to be done).

ATTENTION: unlock phone before connection to PC, if phone is locked auto connection/mount is not happen!!!!

it works for any device - Nexus 4, Nexus 5, Nexus 7, .... that use MTP protocol.

Only one minor nuance is noticed: Nautilus does not copy files/folders if you use Tabs in Nautilus, workaround just use separate nautilus windows.

Thursday, November 28, 2013

Sevntu Checkstyle release 1.10.0

official page: http://sevntu-checkstyle.github.io/sevntu.checkstyle/#1.10.0

New and noteworthy:

New project! sevntu-checkstyle-sonar-plugin -  Project was created as extension to Sonar Checkstyle Plugin for SonarQube, How to use it is described here in detail and pictures, Done by Ruslan Diachenko. More.

New project! sevntu-checkstyle-idea-extension -  Project was created as extension for Checkstyle IDEA plugin for IntelliJ IDEA, How to use it  is described here in details and pictures, Done by Vadim Panasiuk. More.

Bug fixing and improvements in existing Checks: 

CustomDeclarationOrderCheck -  Inner Classes in methods are ignored. Done by Baratali Izmailov. More.

VariableDeclarationUsageDistanceCheck -  Update to ignore raising violation against block constructions/builders and methods with long list arguments, Done by Baratali Izmailov. More.

ConfusingConditionCheck -  Update to skip raising violations on sequential IF, Done by Vadim Panasiuk. More.

NoNullForCollectionReturnCheck -  Fix for NulPointerException, Done by Ilja Dubinin. More.

ForbidCertainImportsCheck -  additional field as Regexp for specifying ignored imports from forbidden, Done by Daniil Yaroslavtsev. More.

Updates in messages -  Done by Baratali Izmailov, Ilja Dubinin, Vadim Panasiuk. More.



General information:
repository: https://github.com/sevntu-checkstyle/sevntu.checkstyle
mail-list: https://groups.google.com/forum/?hl=en#!forum/sevntu-checkstyle

Monday, November 25, 2013

Saturday, November 23, 2013

How to merge github wiki changes from one repository to another


Case:
User forked a repository on GitHub and wiki pages are also forked, user do changes to wiki, but there is no way to send pull request for wiki changes, but Wiki is just another sub repository in GitHub repo, so admin of original repository have to do merge manually.

Example:
source wiki: https://github.com/VadimPanasiuk/sevntu.checkstyle.wiki.git
target/original wiki: https://github.com/sevntu-checkstyle/sevntu.checkstyle.wiki.git

Source of wisdom for massive squash.

Commands:
git clone https://github.com/sevntu-checkstyle/sevntu.checkstyle.wiki.git
cd sevntu.checkstyle.wiki.git
git fetch https://github.com/VadimPanasiuk/sevntu.checkstyle.wiki.git master:master-Vadim
git checkout master-Vadim

#checkout to last non-Vadim's commit  
git reset --hard 34a2f8e627dd6209f4928b6f66a37368789ab492

# do massive squash of all Vadim's changes
git merge --squash HEAD@{1}          
git commit -m "How to use SevNTU Checkstyle in Intellij IDEA (markdown)"
git checkout master

# cherry-pick newly squashed  commit
git cherry-pick cbe1915a65d80769d56f4fbd9a60dd0b88c8c049    

git push


If wiki was forked from latest state of original wiki, you can do it a bit simpler:
git clone https://github.com/sevntu-checkstyle/sevntu.checkstyle.wiki.git
cd sevntu.checkstyle.wiki.git
git pull https://github.com/VadimPanasiuk/sevntu.checkstyle.wiki.git master
# squashing all Vadim's changes in one 
git push origin master

Friday, November 15, 2013

Free API web services

Very concise list of free Web APIs: http://www.programmableweb.com/apis/directory
It is good start for testing REST API calls in your application.

Execute JMX bean method by jolokia wrapper

official site: http://www.jolokia.org/index.html
Reference: http://www.jolokia.org/reference/html/protocol.html#exec

Configuration in your jvisualvm or jconsole:


Example:
[user@server]$ curl localhost:8080/jolokia/exec/my.app:name=my.SyncJob/doSync

Output:
{"timestamp":1384551703,"status":200,"request":{"operation":"doSync","mbean":"my.app:name=my.SyncJob","type":"exec"},"value":null}

Monday, November 11, 2013

Example of orastat usage to find problematic SQL on Oracle server


In Short:
orastat -ser | egrep -v INACT\|BG
# grab PID number of process , for example 9
orastat -sal 9


Here is logs of steps:

[oracle@server ~]$ orastat -ser | egrep -v INACT\|BG
2013/11/07-17:56:50 orastat | oraver=11.2. oraver_short=11.2 oraver_num=11.2 sqlcmd=sp arg=-ser arg2=
2013/11/07-17:56:50 orastat | ORACLE_SID=MYSID ORACLE_HOME=/opt/oracle/product/11.2.0.3 - SunOS hana 5.10 Generic_147441-12 i86pc i386 i86pc
2013/11/07-17:56:50 orastat | Version=3.53 Host=hana Company=some-company
2013/11/07-17:56:50 orastat | -ser: Session Details

Session      Ses    Ses      Wait    Wait       DB     OS                              Session             Last      Last                 +-UserCommits-+ Rollback Program
SID-Ser#     Type   Status   Class / Seq        Userid-Userid-PID-SPID-Term@Server     Start Time          Call ET   SQL    SQL_ID           Count   /Sec    Count Identifier
------------ ------ -------- ------------------ -------------------------------------- ------------------- --------- ------ ------------- -------- ------ -------- ------------------------
9,33801      USER   ACTIVE   User_I/O/51809     SCHEMA2-sb-1234-7525-unknown@other-server.rev 2013/11/07-17:10:05     8_Sec SELECT 1wrbrdf8qrrx8        0      ?        0 JDBC_Thin_Client      
482,56061    USER*  ACTIVE   Network/13         SYS-oracle-14955-14956-?@localhost          2013/11/07-17:56:50     0_Sec SELECT 9d688j2s309du        0      ?        0 sqlplus@localhost_(TNS_V1-V3)

2013/11/07-17:56:50 orastat | Count of USER session(s) in ACTIVE status: 2
2013/11/07-17:56:50 orastat | There are 50 total sessions (mysid=482 - denoted by *)




[oracle@server ~]$ orastat -sal 9
2013/11/07-17:57:23 orastat | oraver=11.2. oraver_short=11.2 oraver_num=11.2 sqlcmd=sp arg=-sal arg2=9
2013/11/07-17:57:23 orastat | ORACLE_SID=MYSID ORACLE_HOME=/opt/oracle/product/11.2.0.3 - SunOS hana 5.10 Generic_147441-12 i86pc i386 i86pc
2013/11/07-17:57:23 orastat | Version=3.53 Host=hana Company=some-company

Session SQL:
--------------------------------------------------------------------------------

SELECT
  cc.company_id  .....
WHERE
  cc.start$ <= sysdate and sysdate < cc.end$

--------------------------------------------------------------------------------

Session Current Wait State:

SID    DB-User          WaitType St WTSeq  Tmr WTTimSec Wait Event Details
------ ---------------- -------- -- ------ --- -------- -------------------------------------
9      SCHEMA2           RealWait  A  51976  -1        1 P1Raw=0000000000000021 P2Raw=00000000000233F5 P3Raw=0000000000000001 db file sequential read WAITED SHORT TIME file# 33 block# 144373 blocks 1

Session Stats:

                                                           ------------------------------------------Session------------------------------------------
SID-Ser#  OS:Userid-PID-Server     DB:User    Program       Hours  CPUTm Commit PFBlks TScans  MemSorts DiskSorts Con+BuGet PhysReads RHRat  LokWaitTm
--------- ------------------------ ---------- ------------ ------ ------ ------ ------ ------ --------- --------- --------- --------- ------ ---------
9,33801   sb-1234-other-server SCHEMA2     JDBCThinClie    0.8 277617      0      0  41609       175         0 118634411     12000  100.0         0

2013/11/07-17:57:25 orastat | orastat | -sal Done






[oracle@server ~]$ orastat -wr
2013/11/07-17:57:48 orastat | oraver=11.2. oraver_short=11.2 oraver_num=11.2 sqlcmd=sp arg=-wr arg2=
2013/11/07-17:57:48 orastat | ORACLE_SID=MYSID ORACLE_HOME=/opt/oracle/product/11.2.0.3 - SunOS hana 5.10 Generic_147441-12 i86pc i386 i86pc
2013/11/07-17:57:48 orastat | Version=3.53 Host=hana Company=some-company
2013/11/07-17:57:48 orastat | -wr: Wait Reasons - By User and Event Type (Idle Waits Excluded)

DB User    Wait Event Type                          Sessions Waiting
---------- ---------------------------------------- ----------------
SCHEMA2     db file sequential read                                 1

2013/11/07-17:57:48 orastat | Total Session(s) Waiting: 1



[oracle@server ~]$ orastat -wr
2013/11/07-17:57:56 orastat | oraver=11.2. oraver_short=11.2 oraver_num=11.2 sqlcmd=sp arg=-wr arg2=
2013/11/07-17:57:56 orastat | ORACLE_SID=MYSID ORACLE_HOME=/opt/oracle/product/11.2.0.3 - SunOS hana 5.10 Generic_147441-12 i86pc i386 i86pc
2013/11/07-17:57:56 orastat | Version=3.53 Host=hana Company=some-company
2013/11/07-17:57:56 orastat | -wr: Wait Reasons - By User and Event Type (Idle Waits Excluded)

DB User    Wait Event Type                          Sessions Waiting
---------- ---------------------------------------- ----------------
SCHEMA2     db file sequential read                                 1

So problem is found, inadequate sequential read that was caused by problems with missed oracle statistics recalculation.

Wednesday, November 6, 2013

Checkstyle idea - require toString() method for Classes

toString() is required for classes that do logging as error to print context of them-self and around.

it might be reasonable to put in toString() only simple types fields that are provided by setXXXXX().

Johua Bloh idea about toString() for classes - Effective Java 2 book - "Item 10: Always override toString" - http://jtechies.blogspot.com/2012/07/item-10-always-override-tostring.html.

Having useful toString() make logging of context during error a simple and pleasant task.

Questions: What criteria we could follow to make a Check that will not force all Classes to have toString() on domains and critical for logging?

Override toString() everywhere is not practical, so we need figure out distinctive characteristics of Classes that have to do it.
- Class that is POJO
- ....

if somebody have ideas please share them on https://groups.google.com/forum/?hl=en#!forum/sevntu-checkstyle or https://groups.google.com/forum/?hl=en#!forum/checkstyle-devel

Monday, October 28, 2013

Trailing Comment Check false positives

Original investigation of Daniil Yroslavtsev:

Check: http://checkstyle.sourceforge.net/config_misc.html#TrailingComment
It has so much problem to report valid code so I do believe that it is switched off  in all companies configurations.

Other support for that Check: Trailing Comments .

But currently Trailing Comment check warns on the following valid cases:

1.

  A. 
  } catch (Exception e) { // SUPPRESS CHECKSTYLE IllegalCatchExtendedCheck now it part of logic

  B.
  } catch (Exception e) { // NOPMD

  C.
  } catch (Exception e) { // NOSONAR

2.
double emerging = 100 - regions.sumEstimate(
  "54138", // Not Applicable/Disclosed
  "19668", // United States
  "200005", // Canada
...
);
ATTENTION: it is not false-positive, as all this magic numbers
 should be moved to constants with proper names



3.


ReportValidationResult r = validationResult.get(i);
  sb.append(
    MessageFormat.format("  report_id={0} (co_id={1}), report_start$=''{2}'', report_end$=''{3}''"
     + ", period_start_date={4}, period_end_date={5}: "
     + "total percent on ''{6}'' is {7}% .... ",
     safeToPlainString(r.reportId), // 0
     safeToPlainString(r.companyId), // 1
     formatTimestamp(r.reportStartDate), // 2
     ...
     formatTimestamp(r.periodEndDate), // 7
     ...
     )

4.

  private static final Color[] COLORS = {             
            new Color(255, 128, 128), // pink
            new Color(255, 255, 128), // yellow             
            new Color(128, 255, 128), // green             
            new Color(128, 255, 255), // cian             
            new Color(255, 128, 192), // margent                      
            new Color(255, 128, 64), // brown
            new Color(0, 128, 128), // blue/gray
            new Color(128, 128, 255), // purple
            new Color(0, 128, 0), // dark green             
            new Color(128, 128, 0), // greenish/grayish     };

5
    record[++columnIndex] = geoAreaEntry.getKey(); // _NAME
    record[++columnIndex] = "unknown"; // _REGION
    record[++columnIndex] = "0"; // _WEIGHT
    record[++columnIndex] = "1"; // _CONF



To fix false-positives we can update Trailing Commentary Sonar check to ignore: //NOSONAR, //SUPPRESS CHECKSTYLE, //NOPMD, // formatter:off, // formatter:on commentaries (with all their possible variants) and other 'valid' comments by regexp in it`s configuration.

But I think, that Checkstyle Trailing Comment check should be disabled for all projects as too many comment formats can be considered as 'valid' and result 'ignore' regexp will be too complex and unreadable.

Monday, October 21, 2013

Checkstyle Check to detect System.out.println usage in java code

Examples of usage in code:
1.
import static java.lang.System.out;
...
out.println("hello!")

2.
System.out.println("hello");




Requirement:
- Analyse import for short name usage "import static java.lang.System.out;"

Possiblle name for Check: ForbidQualifierUsage

Options:
forbiddenFullQualifierNames : String List-  qualifier that we will search for (Example:"System.err.println")
ignoredClasses : Regex -  ignores (Example:"com.mycompany.console.Main")

Bad sides:
1) we can not distinguish methods by parameters so any user method with same name and without package usage will be false-positive if not all methods are investigated to detects overlap of static import in local declaration .

import static java.lang.System.setIn;


public static void setIn(InputStream in) {
    // just a method with similar signature that is compiled
    // if you comment out that method System.setIn will be used (tested in Intelij Idea)
}

public static void main(String... args) {
    setIn(null); // it is local if local method is defined
}

2) Any overloaded methods(same name but different parameters) will be false-positives or known limitations :) :

   System.out.println("hi", "hi")

https://docs.oracle.com/javase/specs/jls/se7/html/jls-7.html#jls-7.5.3

It is permissible for one single-static-import declaration to import several fields or types with the same name, or several methods with the same name and signature. 
So JDK is OK with multiple imports by single static import, so overloaded identifiers are ok.


Resolution:
So it is not complete solution, kind of same level of false positives as RegExp search on code approach.
Could we close eyes on that false-positives?
do you have other ideas and proposal?
https://github.com/checkstyle/checkstyle/issues?q=is%3Aopen+is%3Aissue
Or discussion mail-list - https://groups.google.com/forum/#!forum/checkstyle-devel

Thursday, October 17, 2013

Scala Books

List of books is here: http://www.scala-lang.org/documentation/books.html

I highly recommend to read "Programming in Scala Second edition" book , first edition could be find easily for free download.
This book is much better then "Scala for the Impatient" and "Scala in Depth" as it describe in details functional patterns and basics and have very good history of Scala language and show from where Scala grab ideas and why syntax enhancements are not in Java and .... .

Logging exception message in log event message

When exception happen and you want to log it and the right way to do it is

log.error("Processing is failed during phase {}", phase , e);

In log you will get something like this:
[main] ERROR test.Test - My message Error.
java.lang.RuntimeException:  Long long Message
 at test.Test.main(Test.java:10)

but this approach works fine for logging to output and file and have one problem with reporting to SysLog.
SysLog is not a java logger and it teat each line as separate event so printing stacktrace of exception will result into numerous event on SysLog level - so to resolve problem we use "throwableExcluded=true" , but we loose exception message in same time and it do contains useful information .
Exception message does not disclose reason of problem but it allow to quickly categorize problem and some time it is all that you need to fix a problem.

Example:
        <appender name="SYSLOG" class="ch.qos.logback.classic.net.SyslogAppender">
                <facility>LOCAL1</facility>
                <syslogHost>localhost</syslogHost>
                <suffixPattern>[%d${common.log.date.format}] [%t] [%c{0}] [%p]: %m%n</suffixPattern>
                <throwableExcluded>true</throwableExcluded>
                <filter class="ch.qos.logback.classic.filter.ThresholdFilter">
                      <!-- deny all events with a level below ERROR -->
                      <level>ERROR</level>
                </filter>                                                                                                                            
        </appender>


Alternative way of logging - add Exception message to message of event.

logger.error("Handler execution resulted in exception: " + e.getMessage(), e);
log.error("Processing is failed during phase {} with exception message='{}' ", phase, e.getMessage(), e);

result in message duplicate(see bold text) in file and system output that complicate log reading. This example is not very representative but imagine long message like Spring give on exception.
Example:
[main] ERROR test.Test - My message Error: Long long Message
java.lang.RuntimeException:  Long long Message
 at test.Test.main(Test.java:10)

but this will resolve problem of SyslogAppender and with "throwableExcluded"=true Syslog will have informative message.
Second approach is highly NOT recommended, as it brake nature of log events in java and do duplication of error details.

Throwable class in Java have getMessage() method:  http://docs.oracle.com/javase/7/docs/api/java/lang/Throwable.html

Proposal:
So we need to update SyslogAppender to have "throwableMessageIncluded" or "throwableMessageAppended" parameter that will attach to Event message exception message.
Posted as - http://mailman.qos.ch/pipermail/logback-dev/2013-October/009240.html

Logback resources:
Discussion thread : http://mailman.qos.ch/pipermail/logback-dev/2013-October/009240.html,
or create issue: http://jira.qos.ch/browse
sources only: https://github.com/qos-ch/logback

Monday, October 7, 2013

Meetup with Scala IDEA plugin team leader


It was sad to see that JetBrains team leader use Windows for development:

How to create Checkstyle analog for Scala in IDEA IDE:

video will be there: https://thenewcircle.com/s

Thursday, September 19, 2013

SevNTU Checkstyle version 1.9.0

SevNTU Checkstyle Release 1.9.0 (18/September/2013):

Github: https://github.com/sevntu-checkstyle/sevntu.checkstyle
About project and previous releases: http://sevntu-checkstyle.github.io/sevntu.checkstyle/
To Discuss: https://groups.google.com/forum/?hl=en#!forum/sevntu-checkstyle

New in 1.9.0:

AvoidConstantAsFirstOperandInConditionCheck -  Avoid Constant As First Operand In Condition. If comparing values, C(C++) developers prefer to put the constant first in the equality check, to prevent situations of assignment rather than equality checking. But in Java, in IF condition it is impossible to use assignment, so that habit become unnecessary and do damage readability of code.
Example:
if (null == varialble)"
should be
if (varialble == null)"

Done by Sergey Burtsev.  More


EitherLogOrThrowException -  Either log the exception, or throw it, but never do both. Logging and throwing results in multiple log messages for a single problem in the code, and makes problems for the support engineer who is trying to dig through the logs. This is one of the most annoying error-handling antipatterns.
Example:
       catch (NoSuchMethodException e) {
          LOG.error("Blah", e);
          throw e;
      }
     

Done by Baratali Izmailov.  More


ForbidWildcardAsReturnTypeCheck -  Do not use wildcard types as return types. Rather than providing additional flexibility for your users, it would force them to use wildcard types in client code. Properly used, wildcard types are nearly invisible to users of a class. They cause methods to accept the parameters they should accept and reject those they should reject. If the user of a class has to think about wildcard types, there is probably something wrong with the class’s API.
Example:
public <T> List<? extends T> getValues()

Done by Baratali Izmailov.  More


NoNullForCollectionReturnCheck -  Check detect when method, that must return array or collection, return null value instead of empty collection or empty array.
Example:
public List<String> metValues() { return null; }

Done by Ilja Dubinin.  More


ConfusingConditionCheck -  This check prevents negation within an "if" expression if "else" is present.
For example, rephrase:
if (x != y) smth1(); else smth2(); 
as:
if (x == y) smth2(); else smth1(); 

Done by Vadim Panasiuk.  More

NoMainMethodInAbstractClassCheck -  Forbids main methods in abstract classes. Rationale: existance of 'main' method can mislead a developer to consider this class as a ready-to-use implementation.
Example:
         abstract class AbstractUiJob {
           ...  
             public static void main(String args[]){ ... } 
           ... 
         }
  

Done by Baratali Izmailov.  More

SimpleAccessorNameNotationCheck -  Check only direct fields and setter and getter. Options: 1) member prefix "m_" 2) check only simple getter/setter XXXType getXXXName(); void setXXXName(XXXType).
Example:
private String value; 
void setValue(String val) { this.value = val;}

Reimplemented and fixed by Ilja Dubinin.  More


AbbreviationAsWordInNameCheck -  problem detection of 2 char abbreviation with length limit 1 was fixed.
Example:
int scaleX;

Done by Baratali Izmailov.  More

Thursday, September 12, 2013

Github need to have search of files feature in repo

I will be lovely if Github create search in repo at least by file name (with support of matching by Camel case), it will be very useful during sources investigation. Developers always search for files, and gihub is not just a web hosting - it is online editor, and allow contribution to project without clone to local PC.

This feature exists - https://github.com/blog/793-introducing-the-file-finder just press "t" and search appear.

Scala Lift steps how to compile and run


Lift: http://liftweb.net/download
Examples: https://github.com/dpp/simply_lift

git clone https://github.com/dpp/simply_lift
cd simply_lift/samples/http_rest
./sbt
> update
> jetty-run

But this project use old version of scala and lift and sbt

to do Project from scratch with all explanations: http://cookbook.liftweb.net/#InstallAndRunning
just Rest example: http://simply.liftweb.net/index-5.3.html#toc-Section-5.3

Experiments:
#udpate sbt-launch.jar from http://www.scala-sbt.org/release/docs/Getting-Started/Setup.html to latest version

# remove old file to activate changes
rm project/build.properties

#register eclipse plugin https://github.com/typesafehub/sbteclipse/
echo "addSbtPlugin("com.typesafe.sbteclipse" % "sbteclipse-plugin" % "2.3.0")" > simply_lift/samples/http_rest/project/plugins.sbt


Deploy (jetty, tomact ):
#create a war in target folder
sbt package

#deploy war to Java application server (jetty, tomcat, ...)
https://groups.google.com/forum/#!topic/liftweb/503eibvCvV4

#examples of prod configurations
https://www.assembla.com/spaces/liftweb/wiki/Deployment

Scala vs Nodejs: http://blog.nelsonsilva.eu/2010/12/14/go-vs-scala-vs-nodejs/index.html#disqus_thread

Monday, September 9, 2013

Scala meetup at Netflix office 2013-09-09

How to run scala as script

1. Through interpreter
$ cat script.scala
println("Hello, World!")

Run:
scala script.scala

2. Thought a Scala script

$ cat script.scala
#!/usr/bin/scala!#
println("Hello, World!")

Run as:
$ ./script.scala

3. Compile and then run compiled class
, source.

$ cat script.scala
object Greeting {
    def main(args: Array[String]) = println("Hello world!")
}

Run as:
$ scalac script.scala
$ scala Greeting


Happy scripting!

Friday, September 6, 2013

Comparison web framework for rest api on scala

I did search for the best way to implement simple RESTfull Api service on JVM based technologies (not a node.js and "go" lang). Scala come up as good alternative, additional search shown a Lift framework that looks like popular between Scala community and a lot of posts recommend to do rest api on it, but ...

I met Alexy Khrabrov on one of Scala meetup, and he recommended me to take a look at project https://github.com/Versal/scamper , that do show comparison of RESTful libraries and frameworks, and results are pretty interesting and shown that Lift is not ideal for performance demanding web services.

Just Java Servlet Api 3.0 show petty good performance - that make me think "Should I stay with Java? What can I benefit from Scala ?" :).

Useful stack-overflow discussions: discussion1, discussion2.

My final decision for deep investigation and try is spay: http://spray.io/blog/2013-05-24-benchmarking-spray/

Opposite results of performance: http://irregularexpression.blogspot.com/2012/02/expressjsnodejs-vs-sprayakka-restful.html?m=1

Magic performance of Servlet Api 3.0 is due to "Java NIO (New I/O) API": http://www.onjava.com/lpt/a/5127

Sync local git repository with remotes github and bitbucket

It is common case when you need sync sources between few remote repositories of git, mostly one of them is backup mirror or you have two communities and do not want to bother them what is better github or bitbucket.

#recheck what remotes you have already
git remote

# add link to github and bitbucket remote repo
git remote add github git@hithub.com:my-repo-name.git
git remote add bitbucket https://bitbucket.org/user/my-repo-name
git push -u github master
git push -u bitbucket master

#Somebody sent you fixes to bitbucket rep - pull it from bitbucket and push github
git pull bitbucket
git push github

Saturday, August 31, 2013

Exception logging: 'log and rethrow' vs 'wrap and throw' vs 'log context and re-throw'

If Exception happen during any SQL call to DB server - Where it is good to catch and log it and how to do it?

There is no exact answer, but there are few ways to do it, and some of them have problems that you need to consider.

Case "log and re-throw":
} catch (SQLException e) {
               log.error("Exception occurred ,for sql [" + sql + ]", e);
               throw e;
}

Case "wrap and throw":
} catch (SQLException e) {
           throw new XXXXXDataAccessException("Exception occurred ,for sql [" + sql + ]", e) ;
}

Case "log context and rethrow":

} catch (SQLException e) {
               log.warn("Exception is appear, stack-trace see further in logs, failed sql is [" + sql + ]");
               throw e;
}

Case "log and re-throw":
In log and throw - log with Error or with Warning ?
If we do it with ERROR level - you will brake good pattern "one event - one ERROR message in logs", link to read.
If you will log with WARN - you will spoil logs with duplicated print of stacktrace. On moment of logs review it will not be clear (without review of sources) either exception happen twice or it was logged twice or ....   

General rules for logging and re-throwing:
 - if you develop Library - never log with ERROR level - library does not know how it is used so he have to return all to application and application decide how and when to log it, or even ignore it. Such problem existed in Hibernate, Hibernate, Hibernate, Hibernate, and looks like resolved in Hibernate 4.X only.
- if you are in Application - do logging on level that do know how to treat/translate/process that exception and make exception part of business logic(your algorithm).

Case "wrap and throw":
If you do not log sql and trow exception up, upper level does not have information what sql was in call. You can not log sql with ERROR level at place where you have SQL as is SQL is not an ERROR , exception is a ERROR event - log and rethrow is also bad practice (link to read). Wrapping to your exception make you have custom exception objects and ask yourself and Google why there is no such standard exception - it is so obvious (ones you end up in creation set of custom exceptions in each application).

if you so like to follow "big brothers" - non of these exceptions classes store SQL:
for 15 year off massive SQL usage in java non of them wanted to have it in class as member, because SQL is only one part of problem not a exact reason. 
Non of them have sql as member in class:
http://docs.oracle.com/javase/6/docs/api/java/sql/SQLException.html
https://github.com/hibernate/hibernate-orm/tree/master/hibernate-core/src/main/java/org/hibernate/exception


Case "log context and rethrow":
So logging with WARN is ok, but smb could yell "Error should encapsulate all details of it", and here is nuance - you will never put in Exception object all context, even previous events in logs is a context as you need to see what application did before ERROR. So I recommend to log context of SQL and parameters if any with WARN level and do rethrow exception without wrap.

Main idea is - as you got exception and that level can not log it or process it properly - just print context of exception with WARN level an re-throw it further. Context could be anything: all set of class members, any local variables, whatever will be useful to know state of class when exception happen.

Make a rule for yourself to search for exception in logs and then view events before ERROR event to see a context of problem not only be cause you did logging of exception context but due to the fact that previous events are also context , and they could be even more informational and helpful than smth that you did.

Attention: You can not fix a problem by just looking at exception stacktrace and its message - you need wide context of what happen in application before!

Friday, August 30, 2013

Moving Checkstyle sources from sourceforge to github

Sources: http://sourceforge.net/p/checkstyle/code/ci/default/tree/
Destination: https://github.com/checkstyle

Follow installation steps from "How to convert mercurial to git" manual:  http://codeimpossible.com/2011/12/29/Moving-your-mercurial-repository-to-git/

Command to do clone and convert in same time:
git-hg clone http://hg.code.sf.net/p/checkstyle/code checkstyle-git

Review all participants to find duplicates:
git log --raw | grep "^Author: " | sort | uniq -c

2 Author: Bewied <BenWiederhake.......de>
4 Author: BurnOl <BurnOl@KAZ-027469>
1 Author: convert-repo <devnull@localhost>
1 Author: Lars Kühne <lkuehne>
683 Author: lkuehne <devnull@localhost>
43 Author: mstudman <devnull@localhost>
1054 Author: oburn <devnull@localhost>
9 Author: Oleg Sukhodolsky <o_sukhodolsky>
3 Author: Oliver Burn Cygwin <devnull@localhost>
135 Author: Oliver Burn <devnull@localhost>
288 Author: o_sukhodolsky <devnull@localhost>
410 Author: rickgiles <devnull@localhost>
14 Author: tschneeberger <devnull@localhost>


Create bash file that do squash of authors duplicates - change_names.sh:
#!/usr/bin/bash
   
git filter-branch --tag-name-filter cat --commit-filter '
      if [ "$GIT_AUTHOR_NAME" = "BurnOl" ] && [ "$GIT_AUTHOR_EMAIL" = "BurnOl@KAZ-027469" ]; then
              GIT_AUTHOR_NAME="Oliver Burn";
              GIT_AUTHOR_EMAIL="oliver@.......com";
              GIT_COMMITTER_NAME="Oliver Burn";
              GIT_COMMITTER_EMAIL="oliver@.......com";
              git commit-tree "$@";
      elif [ "$GIT_AUTHOR_NAME" = "convert-repo" ] && [ "$GIT_AUTHOR_EMAIL" = "devnull@localhost" ]; then
              GIT_AUTHOR_NAME="Oliver Burn";
              GIT_AUTHOR_EMAIL="oliver@.......com";
              GIT_COMMITTER_NAME="Oliver Burn";
              GIT_COMMITTER_EMAIL="oliver@.......com";
              git commit-tree "$@";
      elif [ "$GIT_AUTHOR_NAME" = "lkuehne" ] && [ "$GIT_AUTHOR_EMAIL" = "devnull@localhost" ]; then
              GIT_AUTHOR_NAME="Lars Kühne";
              GIT_AUTHOR_EMAIL="lakuehne@.......com";
              GIT_COMMITTER_NAME="Lars Kühne";
              GIT_COMMITTER_EMAIL="lakuehne@.......com";
              git commit-tree "$@";
      elif [ "$GIT_AUTHOR_NAME" = "Lars Kühne" ] && [ "$GIT_AUTHOR_EMAIL" = "lkuehne" ]; then
              GIT_AUTHOR_NAME="Lars Kühne";
              GIT_AUTHOR_EMAIL="lakuehne@.......com";
              GIT_COMMITTER_NAME="Lars Kühne";
              GIT_COMMITTER_EMAIL="lakuehne@.......com";
              git commit-tree "$@";
      elif [ "$GIT_AUTHOR_NAME" = "mstudman" ] && [ "$GIT_AUTHOR_EMAIL" = "devnull@localhost" ]; then
              GIT_AUTHOR_NAME="Michael Studman";
              GIT_AUTHOR_EMAIL="mstudman@.......com";
              GIT_COMMITTER_NAME="Michael Studman";
              GIT_COMMITTER_EMAIL="mstudman@.......com";
              git commit-tree "$@";
      elif [ "$GIT_AUTHOR_NAME" = "Oliver Burn" ] && [ "$GIT_AUTHOR_EMAIL" = "devnull@localhost" ]; then
              GIT_AUTHOR_NAME="Oliver Burn";
              GIT_AUTHOR_EMAIL="oliver@.......com";
              GIT_COMMITTER_NAME="Oliver Burn";
              GIT_COMMITTER_EMAIL="oliver@.......com";
              git commit-tree "$@";
      elif [ "$GIT_AUTHOR_NAME" = "Oliver Burn Cygwin" ] && [ "$GIT_AUTHOR_EMAIL" = "devnull@localhost" ]; then
              GIT_AUTHOR_NAME="Oliver Burn";
              GIT_AUTHOR_EMAIL="oliver@.......com";
              GIT_COMMITTER_NAME="Oliver Burn";
              GIT_COMMITTER_EMAIL="oliver@.......com";
              git commit-tree "$@";
      elif [ "$GIT_AUTHOR_NAME" = "oburn" ] && [ "$GIT_AUTHOR_EMAIL" = "devnull@localhost" ]; then
              GIT_AUTHOR_NAME="Oliver Burn";
              GIT_AUTHOR_EMAIL="oliver@.......com";
              GIT_COMMITTER_NAME="Oliver Burn";
              GIT_COMMITTER_EMAIL="oliver@.......com";
              git commit-tree "$@";
      elif [ "$GIT_AUTHOR_NAME" = "Oleg Sukhodolsky" ] && [ "$GIT_AUTHOR_EMAIL" = "o_sukhodolsky" ]; then
              GIT_AUTHOR_NAME="Oleg Sukhodolsky";
              GIT_AUTHOR_EMAIL="os97673@.......com";
              GIT_COMMITTER_NAME="Oleg Sukhodolsky";
              GIT_COMMITTER_EMAIL="os97673@.......com";
              git commit-tree "$@";
      elif [ "$GIT_AUTHOR_NAME" = "o_sukhodolsky" ] && [ "$GIT_AUTHOR_EMAIL" = "o_sukhodolsky" ]; then
              GIT_AUTHOR_NAME="Oleg Sukhodolsky";
              GIT_AUTHOR_EMAIL="os97673@.......com";
              GIT_COMMITTER_NAME="Oleg Sukhodolsky";
              GIT_COMMITTER_EMAIL="os97673@.......com";
              git commit-tree "$@";
      elif [ "$GIT_AUTHOR_NAME" = "o_sukhodolsky" ] && [ "$GIT_AUTHOR_EMAIL" = "devnull@localhost" ]; then
              GIT_AUTHOR_NAME="Oleg Sukhodolsky";
              GIT_AUTHOR_EMAIL="os97673@.......com";
              GIT_COMMITTER_NAME="Oleg Sukhodolsky";
              GIT_COMMITTER_EMAIL="os97673@.......com";
              git commit-tree "$@";
      elif [ "$GIT_AUTHOR_NAME" = "rickgiles" ] && [ "$GIT_AUTHOR_EMAIL" = "devnull@localhost" ]; then
              GIT_AUTHOR_NAME="Rick Giles";
              GIT_AUTHOR_EMAIL="rickgiles";
              GIT_COMMITTER_NAME="Rick Giles";
              GIT_COMMITTER_EMAIL="rickgiles";
              git commit-tree "$@";
      elif [ "$GIT_AUTHOR_NAME" = "tschneeberger" ] && [ "$GIT_AUTHOR_EMAIL" = "devnull@localhost" ]; then
              GIT_AUTHOR_NAME="Travis Schneeberger";
              GIT_AUTHOR_EMAIL="leo.herbie@.......com";
              GIT_COMMITTER_NAME="Travis Schneeberger";
              GIT_COMMITTER_EMAIL="leo.herbie@.......com";
              git commit-tree "$@";
      else
              git commit-tree "$@";
      fi' HEAD



explanation to script:
" --tag-name-filter cat" - required to not loose tags after history rewrite.
"GIT_COMMITTER_NAME and GIT_COMMITTER_EMAIL" - to let github show commits correctly at page of commits history, in other case you have invalid-email-address in details for each commmit.

Do changes to authors and do push to remote:
./change_names.sh
git remote add origin git@github.com:checkstyle/checkstyle.git
git push -u origin master --tags

Sources of wisdom:
In filter-branch shell syntax for IF-ELSE should be used - http://www.dreamsyssoft.com/unix-shell-scripting/ifelse-tutorial.php
usage of bash syntax for IF- ELSE cause errors - http://blog.linuxacademy.com/linux/conditions-in-bash-scripting-if-statements/
http://git-scm.com/book/ch6-4.html (chapter "Changing E-Mail Addresses Globally")
http://ayende.com/blog/4532/extracting-a-list-of-committers-from-git

Monday, August 26, 2013

Rest api development best practices resources

Here is just list of resources that I found and do not want search one more time:

Разработка web API / Хабрахабр
http://habrahabr.ru/post/181988/

Designing a Beautiful REST+JSON API (and other video around)
http://www.youtube.com/watch?v=5WXYw4J4QOU

steps toward the glory of REST
http://martinfowler.com/articles/richardsonMaturityModel.html

API creation blog:
https://blog.apigee.com/
http://apigee.com/about/api-best-practices/all/ebook

API creation mail list:
https://groups.google.com/forum/#!forum/api-craft

Tuesday, August 13, 2013

Krusader have problem with unpacking tag.gz

It is not a first time I noticed that after unzip/unarchive/unpacksome archive some files are missed. But when you do it from terminal or ubuntu befault archive manager - all works fine.

issue was reported: https://bugs.kde.org/show_bug.cgi?id=323472

Sunday, July 28, 2013

Extract certain tag with inner tags from huge XML

Task: You have huge XML file (300 Mb), you need to get out(filter out/extract/grab) it only one tag with specific value.

Under Ubuntu 12.04 install xslt processor:
sudo apt-get install xsltproc

Structure of huge XML (hugeFile.xml):
<?xml version="1.0" encoding="ISO-8859-1"?>
<Feed ExtractDate="07/25/2013" ExtractTime="15:30:15">
  ..... a lot of companies information
<COMPANY ... LegalName="MyCompany" .....>
     ..... a lot of inner tags .....
  </COMPANY>
 ..... a lot of companies information
</Feed>
 

Create extract.xsl file:
<?xml version="1.0" encoding="utf-8"?>
<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">
  <xsl:template match="/">

<xsl:element name="TagToExtract">
   <xsl:apply-templates select="//COMPANY[@LegalName='MyCompany']" />
</xsl:element>

 </xsl:template>
  <xsl:template match="//COMPANY[@LegalName='MyCompany']">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()" />
    </xsl:copy>
  </xsl:template>
  <xsl:template match="@*|node()">
    <xsl:copy>
      <xsl:apply-templates select="@*|node()" />
    </xsl:copy>
  </xsl:template>
</xsl:stylesheet>

Execute XSLT processor (12 seconds):
xsltproc extract.xsl hugeFile.xml > 1.xml



Saturday, July 27, 2013

Keep command launching after logout from terminal

It is very common case - you need to login to linux box, launch smth huge and that takes time, but you do not want to wait its finish and want to logout, or you afraid that internet connection could be broken in the middle of process that terminate your process.

thanks a lot for explanation to Dennis Williamson at here.

Two scenarios:
1) You launch process in background:

$ nohup myprogram &
$ logout

Explanation: nohup is allow process to ignore termination signal SIGHUP (“hang-up”)  on your logout (manual), "&" - just launch process in background and do not block current terminal session.


2) You launched process but forget to do it in background:

$ myprogram
<press Ctrl+Z>
[1]+  Stopped     myprogram
$ disown -h %1
$ bg 1
[1]+ myprogram &
$ logout

Explanation (copy-paste from stackoverflow):
You press ctrl-Z. The system suspends the running program, displays a job number and a "Stopped" message and returns you to a bash prompt. You type the "disown -h %1" command (here, I've used a "1", but you'd use the job number that was displayed in the "Stopped" message) which marks the job so it ignores the SIGHUP signal (it will not be stopped by logging out). Next, type the "bg" command using the same job number. This resumes the running of the program in the background and a message is displayed confirming that. You can now log out and it will continue running...
 ...You should be aware that when you use the "bg" command the result is the same as if you'd run your program in the background with an ampersand (&). It won't have any output to stdout so it should be made to write output to a file (nohup will redirect standard output to nohup.out or ~/nohup.out if you don't redirect it yourself).

FYI: to resume to stopped process to foreground you can use "fg" of "fg <number>" in our case "fg 1", where "<number>" - is number that shown after Ctrl+z.

Wednesday, July 10, 2013

How to resolve system variables in spring xml configuration

during Tomcat launch you can specify system variable as ".... -Dmy.value=Hello .... "

but do not forget to put in you Spring configuration following bean:
<bean id="system-properties" class="org.springframework.beans.factory.config.PropertyPlaceholderConfigurer" />

to make this variable resolvable in any other bean configuration like:
...

<entry key="myValue" value="${my.value}" />
..

Friday, July 5, 2013

Remove job from Oracle

There are few types of jobs:
till Orecle10 - packages DBMS_JOB is used.
from Oracle11 - package DBMS_SCHEDULER is used.

DBMS_JOB are used numbers for identifier.
DBMS_SCHEDULER use valid oracle object name (that can not be started from numbers) 
Links to spec: dbms_job is deprecatedschedulerdbms_joball_jobs.

when you try to remove old job by new package you got (2364 is job identifier):

Error starting at line 1 in command:
BEGIN
  DBMS_SCHEDULER.DROP_JOB('2364');
END;
Error report:
ORA-20001: comma-separated list invalid near ,
ORA-06512: at "SYS.DBMS_UTILITY", line 236
ORA-06512: at "SYS.DBMS_UTILITY", line 272
ORA-06512: at "SYS.DBMS_SCHEDULER", line 623
ORA-06512: at line 2 



From SQLDeveloper I got different error when use old package DBMS_JOB:
DBMS_JOB.remove(2364);

[ERROR] ORA-23421: job number 2364 is not a job in the job queue
[ERROR] ORA-06512: at "SYS.DBMS_SYS_ERROR", line 86
[ERROR] ORA-06512: at "SYS.DBMS_IJOB", line 770
[ERROR] ORA-06512: at "SYS.DBMS_JOB", line 180
[ERROR] ORA-06512: at line 2

But it was ok from sqlplus under owner user :

BEGIN DBMS_JOB.remove(2364); END; commit;

How to change location of monitor according to laptop

When you use additional monitor to you laptop it is good to have ability to specify there is you monitor is located in relation to laptop to use mouse movement easily.

By default Ubuntu put your laptop to left side and monitor to the right side, so moving mouse to right edge will move it next monitor. That is cool, but what if you monitor is located on left side of your laptop,  and you can not switch monitor physical location - you need to switch its location  virtually :).

launch "Displays" or run "System Settings" and click on "Displays" in settings. You will see layout like this:

Just grab monitor rectangle by mouse and move it to left side of laptop's screen rectangle. That is all.

move window to other monitor at ubuntu



You need to install Compiz Config Settings Manager:
"sudo aptitude install compizconfig-settings-manage"

and launch it from Dash(main button) by filtering "compiz".

select in the "Window Management" section a "Put" item. With that enabled, you can set a shortcut key in the "bindings" tab for "Put to Next Output". By default, it's disabled, but is easy to set.
No restart is required (at least on Ubuntu 12.04 x64).

If you have two monitors - this combination will work like swap of window location on monitors.

My default is "Cntl+~".

Thursday, June 27, 2013

MOZILLA_FIVE_HOME exception while opening Checkstyle configuration


During opening Checkstyle configuration Ubuntu 12.04 x64 user can experience following error:

Unhandled event loop exception
No more handles [Unknown Mozilla path (MOZILLA_FIVE_HOME not set)]

Solution:

sudo apt-get install libwebkitgtk-1.0-0




There is a bug reports for eclipse.

Sunday, June 9, 2013

Listen HitFM on Linux (Ubuntu and Fedora )

Listening HitFm from web page is not convenient as it force to open new browser window - that make inconvenience for window switching and .... .

1. Get URL for using it your Radio listen application
http://www.hitfm.ua/HitFM.m3u

Note:
if you have problem with that url , it myght be that HitFm changed it, so go to http://www.hitfm.ua/ press "Слушать радио", in newly appeared window look at links below "Слушайте Хiт FM у себя на компьютере:" copy URL from "Быстрый интернет (128 kb/s)"

2. I used default application in Ubuntu and Fedora - Rythmbox.
Add new radio by pressing "New Internet Radio Station ..." in context menu over "Radio" item in Library (left panel).

3. press "Play".

Note:
For Fedora be sure that you installed support for mp3 format, it is not installed by default - steps to install.
For Ubuntu you need to install Restricted codes(mp3,.....) " sudo apt-get install ubuntu-restricted-extras".

Wednesday, June 5, 2013

Data conversion and mining tool of any text, CSV, JSON fomats


Google tool : http://openrefine.org/

Video explain all .... this is very cool tool for data mining, and data transforming. I usually use LibreOffice Calc for this data mining, that tools rocks as it could:
- conditions for columns creation from other column
- language for data minding from columns
- recording of data translation to repeat again and again
- usage of web services to generate new column base on existing column
- embedded language to parse JSON
- .... just look at video ...

Bulk files rename in linux


Task: in folder /www/logs/jobs/ rename all files started from "rp-repgen" to "repgen", in other words change prefix of files.

Solution:
 for file in /www/logs/jobs/rp-repgen*; do dest="${file/rp-repgen/repgen}";mv $file $dest; done

Wednesday, May 1, 2013

How to split repository into two repositories


Source of wisdom.

We have infra repository that is have folder git_tool that is going to be new repository now.

State of repository "infra" before split:
jenkins_checks
svn2git
git_tools
build-release.sh              

Task: move "git_tools" to separate repository "infra_new".

Steps:
1. Create new repo for "git_tools" only
$ git clone  git@git.yourserver.com:infra
$ cd infra/
$ git filter-branch --prune-empty --index-filter 'git rm --cached --ignore-unmatch jenkins_checks/* sb-svn2git/* build-release.sh eclipse-compiler-warnings.epf ' HEAD
$ gitk --all
$ git remote add origin https://git@git.yourserver.org/infra_new.git
$ git push -u origin --all

2. Remove all history remnants from old "infra" repo:

$ git clone  git@git.yourserver.com:infra
$ cd infra/
$ git filter-branch --prune-empty --index-filter 'git rm --cached --ignore-unmatch git_tools/*' HEAD
$ gitk --all
$ git push -u origin --all --force

Thursday, April 25, 2013

Share folder between Windows and Ubuntu 12.04 by mean of Samba

The best way to share folder between different Windows and Linux is to share it on Linux and let Windows explore it in network. Sharing on Windows and exploring from computer and folder from Ubuntu is never stable (Linux guru could do this, but it is not for ordinary user).

Follow pretty good illustrated instructions, works fine for 12.04:
http://www.liberiangeek.net/2010/06/how-to-quickly-change-workgroup-name-in-ubuntu-10-04-lucid-lynx/

Restart samba service after each sharing( just in case :) ), in Ubuntu 12.04 as:

sudo restart smbd

If smth does not work, restart Linux PC, this will help windows to see your Linux :). 

Tuesday, April 23, 2013

Cleaning cache for Apache by htcacheclean


Detecting apache cache problem, cleaning cache for apache by htcacheclean.
good pages: page1, page2, page3.

My case:
in apache config /www/etc/apache/adm/virt.serv/10_my-service.cfg:

CacheRoot /www/cache/my-service
CacheDirLevels 2
CacheDirLength 1
CacheDefaultExpire 3600
CacheIgnoreCacheControl On
CacheEnable disk /my-service/

I get different content:
wget https://server.com/my-service/css/layout.css
wget https://test-server.com/my-service/css/layout.css

as HTTP Header have value:
Cache-Control: max-age=31556926, must-revalidate 
How to get this in Firebug addon of Firefox:

Or you can do this from terminal :
06:20 PM ~ $ curl -I https://server.com/my-service/css/layout.css
HTTP/1.1 200 OK
Date: Wed, 24 Apr 2013 01:20:08 GMT
Server: Apache/2.2.19 (FreeBSD)
Expires: Thu, 24 Apr 2014 07:08:54 GMT
Cache-Control: max-age=31556926, must-revalidate
Last-Modified: Mon, 22 Apr 2013 17:32:10 GMT
Content-Length: 36490
Connection: close
Content-Type: text/css
Set-Cookie: Coyote-2-4a7446e9=c0a89971:0; path=/

So caching could be adjusted on Tomcat side, after then on Apache side(CacheDefaultExpire and "CacheEnable disk /my-service/"), and then kept by user's Browser(Cache-Control: max-age). 

Cache could be adjusted in tomcat side in Spring config (28800 = 8 hour):
<mvc:resources mapping="/css/**" location="/css/" cache-period="28800" />

So after each Tomcat rollout we have to do:
htcacheclean -p/www/cache/my-service -l1K -v

if you have problem on apache clear cache like "An error occurred, cache cleaning aborted."
be sure you  /www/cache/my-service/ is created and "chown apache /www/cache/my-service/", "apache" username is taken from "ps -ef| grep http" as on Freebsd it is "www". Also be sure you launch htcacheclean under right user.