Sunday, February 7, 2016

Master development problem with pictures

we need to decide: Jenkins should test pre-rollout state or post prollout state of sources.

Workflow:  master --> development --> topicBranch
Simple exception stack trace examination from PROD (only require switch to master)
new iteration/feature come with version bump so PROD code could be bumped with mvn-release-plugin without worries to interfere with development.
only 2 branches are required
master is not correspond to PROD after release was roll-backed on PROD!!!!! no way to revert/reset master without pain for all developers (conflicts are unavoidable).
person that do rollout to PROD have to remember to do merge from development to master before rollout to PROD - RomanIvanov always fail to this on time. Rollout could be done in out of working hours by SysAdm, so it impossible to sync that events.
"hot fix branches"( is better than do changes in master and then force rebase on development that could not be that simple. Two master("master","mastr-hotfix") branch appear and same problem is raised "Where is prod code?"
in case of hotfix over "master" Jenkins have no easy ability to test changes before rollout (if you follow a model merge to master after rollout ) ...
any changes to master, with development far in forward, make it impossible to keep Fast forward merge
keep master-1.7-hotfix branch is not good for jenkins as it does not aware of it - so you deploy changes that are not tested.

Problem: we released to RPOD repgen but not a RP, so I can not do merge to master as this two projects lives in same repo - is it git structure problem or problem of your workflow?
Problem: reports 1.3 and 1.4  - we cannot do fast-forward merge as we have conflicts in changes (we did cherry-pick from master to development), we can not do just merge without conflicts, with non fast-forward we will loose all commits history as all of them will come to one merge-commit, tags will be lost as we change SHA1 for commits. Should we do reset of master till common commit and then do fast-forward merge ? should we store all hot fixes ? should we name it like branch and keep in git for a while but in this case how it is different from alternative git workflow ?
Problem: 'themes' - fix was applied to master and then in few days was applied to development, on merge development to master I got conflict, I could resolve it - but in master I will got merge-commit. I could revert that fix in master that will help me to merge without problem but still - one merge commit(history is lost). I did HACK by reset to common commit with development and that allowed me do fast-forward (master keep history).  Resume: looks like to keep history we need to work in master, or treat development branch as the most valuable branch (that never have to be removed or lost).

Delema: how to merge from developement to master?  up to a tag(3.1.6) or next commit after tag ?
in case of "next commit after tag" - it is convenient to do versioning as we stay on SNAPSHOT
in case of "up to tag" - is more correct as we rollouted exact version, it even make more sense as it comes to master-X.X.X-hotfix creation where it will unclear in history why it was required to revert changes from 3.1.7-SNAPSHOT to as in master 3.1.7 have never existed.

Problem: master, master-3.2.12-hotfix, master-3.2-hotfix  (all branches were created as mentioned here) -  master-3.2.12-hotfix contains tag that rollouted on PROD, after rollout from master-3.2-hotfix by tag 3.2.14 we can not remove master-3.2.12-hotfix branch as it contains tag that could be useful in case rolback on PROD.
Command "git merge -s ours master-3.2.12-hotfix" will not help as do exactly what we need from keeping commits/tags in history, but it DO CREATE ADDITIONAL/MERGE commit, that will make problem when it comes to development->master merge.

problem: master ans master-hotfix, master-hotfix depend on DB changes that is incompatible with master code. So we have master that is failing on UTs but master-hotfix is not. So model of checksout "master*" in Jenkins does not help.

!!problem!!: if repository is composite of few projects that could be releases/rollouted/tagged separately (ld and ld-data-job) we can not merge to master all as ld-data-job is rollouted to RPOD, but some changes in ld-server are not rollouted and cannot be merged to master as it brake main idea that master===prod  code.

Workflow : hot-fix branch <-- master --> topicbranch
only 1 common branches is required
not clean when remove hot-fix branches - but will not be a problem
Problem: we didnot bumped minor version in pom till rollout to prod we do few rollouts with build number increase, but still not ready to prod, and now we got PROD critical issue, we come back to prod revision to create branch, but we cannot use maven:release plugin as next few build numbers already occupied by rollouts to dev. Will be jars override in maven secure, can we trust to this ?
How we will secure UT passed for Prod version before rollout to prod ? not sure that developer will remember to goto Jenkins change config to new commit reference and launch configuration.... but I could be not right here .

It is highly required for ease and speed of development and fix delivering to have hot-fix branch to backport some features/fixes and rollout them more frequently, without being blocked by next rollout changes.


How to get statistic info from checkstyle-result.xml

grep "source=" checkstyle-result.xml | sed 's/.*source="//' | sed 's/"\/>//' | sed 's/Check//' | sed 's/' | sort | uniq -c | sort -n

Performance problem while using Checks that extends AbstractJavadocCheck

Get all Javadoc Checks that are using Grammar parser for Javadoc:
~/java/git-others/checkstyle/checkstyle [master|✔] $ grep -l -r --include "*.java" --exclude "*" "extends AbstractJavadocCheck" . | sed -E 's/.\/src\/.*javadoc\///' | sed "s/"

Execution test that show performance degradation in 150 times (19sec vs 5m8sec):

$ grep -E "JavadocTagContinuationIndentation|NonEmptyAtclauseDescription|SingleLineJavadoc|JavadocParagraph|SummaryJavadoc|AtclauseOrder|AbstractJavadoc" google_checks.xml 
        <module name="NonEmptyAtclauseDescription"/>
        <module name="JavadocTagContinuationIndentation"/>
        <module name="SummaryJavadoc">
        <module name="JavadocParagraph"/>
        <module name="AtclauseOrder">
        <module name="SingleLineJavadoc">
$ grep -E "JavadocTagContinuationIndentation|NonEmptyAtclauseDescription|SingleLineJavadoc|JavadocParagraph|SummaryJavadoc|AtclauseOrder|AbstractJavadoc" google_checks_only-one-javadoc-check.xml

        <module name="NonEmptyAtclauseDescription"/>
$ grep -E "JavadocTagContinuationIndentation|NonEmptyAtclauseDescription|SingleLineJavadoc|JavadocParagraph|SummaryJavadoc|AtclauseOrder" google_checks_no-javadoc-checks.xml

$ time java -jar checkstyle-6.15-all.jar -c google_checks_no-javadoc-checks.xml guava/
Starting audit...
Audit done.

real 0m19.532s
user 0m56.985s
sys 0m0.807s

$ time java -jar checkstyle-6.15-all.jar -c google_checks.xml guava/
Starting audit...
Audit done.

real 5m8.913s
user 6m25.283s
sys 0m1.287s

$ time java -jar checkstyle-6.15-all.jar -c google_checks_only-one-javadoc-check.xml guava/
Starting audit...
Audit done.

real 5m7.450s
user 6m14.414s
sys 0m1.179s

The only excuse for such performance degradation is that parsing is done correctly in comparison to RegExp implementation of parsing of javadoc that have anonymous amount of bugs.

This need to be fixed.....

Monday, November 23, 2015

How to uppercase string in linux command line

Presume you have a text:

and you need to convert test to upper case by smth in OS

1) Wrap text with "tr [a-z] [A-Z]<<EOF" and "EOF" at top and bottom respectively and copy to terminal.

tr [a-z] [A-Z]<<EOF


go to browser and do search for any online service that do uppercase: 

Sunday, October 18, 2015

Software Quality Award comment for Checkstyle

ATTENTION: non of following should be treated as appellation or demand to reconsider review of Checkstyle project in Award. This post is targeted only to contributors who asking why Checkstyle missed so obvious points.

> 83K Java LoC, 553K HoC

suseika/inflectible (5K LoC, 36K HoC) - winner
testinfected/molecule     (10K LoC, 43K HoC)
coala-analyzer/coala      (14K LoC, 160K HoC)
xvik/guice-persist-orient (17K LoC, 54K HoC)
raphw/byte-buddy      (84K LoC, 503K HoC)
citiususc/hipster   (5K LoC, 64K HoC)

checkstyle/checkstyle (83K LoC, 553K HoC)
kaitoy/pcap4j         (42K LoC, 122K HoC)

we are second biggest project in Final. But we are the oldest projects, we are almost 15 years old. That point will explain some design problems below.

> There are many ER-ending classes, like SeverityLevelCounter, Filter, and AbstractLoader (for example), which are anti-patterns.

That is controversial anti-pattern and I will explain why there is no damage this in separate post. No comments for this point, it is just philosophy of Award owner. 

> There is a whole bunch of utility classes, which are definitely a bad thing in OOP. They are even groupped into a special utils package, such a terrible idea.

Utility is good as it is stateless realization of algorithms. Mismatch of design philosophy between Author and us.

> Setters and getters are everywhere, together with immutable classes, which really are not an OOP thing, for example DetectorOptions.

Yes, that is effect of 15 years old project and numerous existing Checkstyle's integration and extensions that already exists in this world. We can not changes this - it will be huge compatibility damage. Nobody will benefit from such update. But we will already in progress to make Checkstyle more immutable.
Attention: getter/setter is very controversial anti-pattern (mutability is a concern, but there is not problem in naming), I will explain this in separate post.

> NULL is actively used, in many places — it's a serious anti-pattern

Yes, that problems comes from core Checkstyle - ANTLR2 parser.

> I've found five .java files with over 1000 lines in each of them, for example 2500+ in

Yes, and we do this on good reason. First of all is in test ares, secondly it is generated :).
There are exceptions in rule for big files:
2) UTs are also in exclusion, as it is easier to find from class --> and not puzzle what  mean (and how author decide when test should go second test file). So it is OK.

> There are direct commits to master made by different contributors and some of them are not linked back to any tickets. It's impossible to understand why they were made. Look at this for example: 7c50922. Was there a discussion involved? Who made a decision? Not clear at all.

Such commit is mine :). But we do have strict control of commit message, we are even more fanatic than all other projects. Whole idea is described - . There is no problem with direct commits, especially from the most experienced owners of it (see all details in wiki link).

Releases are not documented at all.

It is first time I see that somebody pay attention to , we do have Release Notes in human friendly way on our main HTML site -  (it is not a list of commits !!!! users do not need commits!!! )

> Release procedure is not automated. At least I didn't find any release script in the repository.

It is automated and done by standard maven procedure "mvn release" no need for special shell file and we are multi-OS project (Linux, Windows, MacOS). 
Release is not a binaries copy to artifact repository!  Here is detailed instructions on how to make a release - with all details of how systems should be prepared and what should be updated after version bump. But I agree, I would be happy to automate it. For now it does not make sense to spend time on difficult automation, if we release ones a month.


some quotes from Award page:
I'm a big fan of object-oriented programming in its purest form
Strict and visible principles of design.

We participated in Pure OOP contest. I do not share fanatic following of pure OOP designs, I am more in favor of changes to be more functional. Any re-factoring in favor of OOP is not possible without braking compatibility with plugins/extensions.
So 7th place is very good.

Friday, October 16, 2015

Code coverage could help to detect dead or extra code

Code coverage could for help detect dead code or useless conditions, see example at

No way to reproduce in UT it mean that code is dead or useless.

Interesting automation for dead code removal base on code coverage (Guantamo project) - 

How to cache maven local repo between Travis builds

caching is allowed only for private repos or public repos that use "Using container-based infrastructure".

1. instruct travis to use "container-based infrastructure"
2. instruct travis what folder you need to cache

Official documentation:

Results: that caching speedup build on ~3-4 minutes for each jdk.