buildbot failure in ASF Buildbot on ofbiz-trunk

[buildbot]

The Buildbot has detected a new failure of ofbiz-trunk on ASF Buildbot.
Full details are available at:
 http://ci.apache.org/builders/ofbiz-trunk/builds/2245

Buildbot URL: http://ci.apache.org/

Buildslave for this Build: isis_ubuntu

Build Reason:
Build Source Stamp: [branch ofbiz/trunk] 895940
Blamelist: jonesde

BUILD FAILED: failed svn

sincerely,
 -The ASF Buildbot

[Adam Heath-2]

[hidden email] wrote:

> The Buildbot has detected a new failure of ofbiz-trunk on ASF Buildbot.
> Full details are available at:
>  http://ci.apache.org/builders/ofbiz-trunk/builds/2245
>
> Buildbot URL: http://ci.apache.org/
>
> Buildslave for this Build: isis_ubuntu
>
> Build Reason:
> Build Source Stamp: [branch ofbiz/trunk] 895940
> Blamelist: jonesde

Scott has said in the past that the buildbot doesn't run on every
checkin.  This means that multiple commits might be tested.  Would it
be possible to have failure messages list all of the revisions that
are being covered, based on the last successful commit?

Additionally, it'd be nice if once a failure is detected, it'd either
do an auto binary disection, to find the commit that actually failed
in the list of covered revisions, or it could just try them all.

For instance, if 891234 is good, and then 892765 fails.  892765 would
get recorded as a state change, putting ofbiz into fail mode.  It
would then try 891562, 891840, 892496, and 892599, reporting that it
tried these other revisions, and noticed the error first occurred at
892496, then ofbiz would stay in fail mode, and the buildbot would
keep quiet until it detects that ofbiz has been fixed.

Or, if that is too complicated, at least list all the covered
revisions, and the user that committed each revision.

[Scott Gray-2]

On 6/01/2010, at 10:19 AM, Adam Heath wrote:

> [hidden email] wrote:
>> The Buildbot has detected a new failure of ofbiz-trunk on ASF  
>> Buildbot.
>> Full details are available at:
>> http://ci.apache.org/builders/ofbiz-trunk/builds/2245
>>
>> Buildbot URL: http://ci.apache.org/
>>
>> Buildslave for this Build: isis_ubuntu
>>
>> Build Reason:
>> Build Source Stamp: [branch ofbiz/trunk] 895940
>> Blamelist: jonesde
>
> Scott has said in the past that the buildbot doesn't run on every
> checkin.  This means that multiple commits might be tested.  Would it
> be possible to have failure messages list all of the revisions that
> are being covered, based on the last successful commit?

This is what happens, for example take a look at the build after the  
one above which covered a few commits: http://ci.apache.org/builders/ofbiz-trunk/builds/2246

> Additionally, it'd be nice if once a failure is detected, it'd either
> do an auto binary disection, to find the commit that actually failed
> in the list of covered revisions, or it could just try them all.
>
> For instance, if 891234 is good, and then 892765 fails.  892765 would
> get recorded as a state change, putting ofbiz into fail mode.  It
> would then try 891562, 891840, 892496, and 892599, reporting that it
> tried these other revisions, and noticed the error first occurred at
> 892496, then ofbiz would stay in fail mode, and the buildbot would
> keep quiet until it detects that ofbiz has been fixed.

I have no idea if buildbot is capable of this but you're more than  
welcome to investigate it and request changes from infra.
In general though I think it's usually fairly obvious what the cause  
of a problem is based on a quick look at the stdio from the failed  
build task.  I don't think it's a huge ask for each committer on the  
blamelist to take a minute or two to figure out if it's their problem  
or not.

>
> Or, if that is too complicated, at least list all the covered
> revisions, and the user that committed each revision.

[Adam Heath-2]

Scott Gray wrote:

> On 6/01/2010, at 10:19 AM, Adam Heath wrote:
>
>> Scott has said in the past that the buildbot doesn't run on every
>> checkin.  This means that multiple commits might be tested.  Would it
>> be possible to have failure messages list all of the revisions that
>> are being covered, based on the last successful commit?
>
> This is what happens, for example take a look at the build after the one
> above which covered a few commits:
> http://ci.apache.org/builders/ofbiz-trunk/builds/2246

Wasn't certain, thought it might, but it's nice to have it confirmed.

>> Additionally, it'd be nice if once a failure is detected, it'd either
>> do an auto binary disection, to find the commit that actually failed
>> in the list of covered revisions, or it could just try them all.
>>
>> For instance, if 891234 is good, and then 892765 fails.  892765 would
>> get recorded as a state change, putting ofbiz into fail mode.  It
>> would then try 891562, 891840, 892496, and 892599, reporting that it
>> tried these other revisions, and noticed the error first occurred at
>> 892496, then ofbiz would stay in fail mode, and the buildbot would
>> keep quiet until it detects that ofbiz has been fixed.
>
> I have no idea if buildbot is capable of this but you're more than
> welcome to investigate it and request changes from infra.

Standard open-source response, kinda expected it.  It'd still be nice
if the request was forwarded on, if it wasn't too much of a hassle.
If it is, I respect that, and I'll just add it to my infinite todo
list squared.

> In general though I think it's usually fairly obvious what the cause of
> a problem is based on a quick look at the stdio from the failed build
> task.  I don't think it's a huge ask for each committer on the blamelist
> to take a minute or two to figure out if it's their problem or not.

Computer time is free, human time is not.  Computers are continually
getting faster, while humans get slower.

Have you had a chance to play with git bisect yet?  It makes me feel
squishy in my happy place.

[Scott Gray-2]

On 6/01/2010, at 10:50 AM, Adam Heath wrote:

> Scott Gray wrote:
>> On 6/01/2010, at 10:19 AM, Adam Heath wrote:
>>
>>> Additionally, it'd be nice if once a failure is detected, it'd  
>>> either
>>> do an auto binary disection, to find the commit that actually failed
>>> in the list of covered revisions, or it could just try them all.
>>>
>>> For instance, if 891234 is good, and then 892765 fails.  892765  
>>> would
>>> get recorded as a state change, putting ofbiz into fail mode.  It
>>> would then try 891562, 891840, 892496, and 892599, reporting that it
>>> tried these other revisions, and noticed the error first occurred at
>>> 892496, then ofbiz would stay in fail mode, and the buildbot would
>>> keep quiet until it detects that ofbiz has been fixed.
>>
>> I have no idea if buildbot is capable of this but you're more than
>> welcome to investigate it and request changes from infra.
>
> Standard open-source response, kinda expected it.  It'd still be nice
> if the request was forwarded on, if it wasn't too much of a hassle.
> If it is, I respect that, and I'll just add it to my infinite todo
> list squared.

The idea is to ask infra to do no more than is necessary (they're  
unpaid volunteers), so at the very least we (you) should find out if  
buildbot is capable of doing what you desire.
As below human time isn't free and I'd prefer it if people deal with  
infra directly if the change is important enough to them.

>> In general though I think it's usually fairly obvious what the  
>> cause of
>> a problem is based on a quick look at the stdio from the failed build
>> task.  I don't think it's a huge ask for each committer on the  
>> blamelist
>> to take a minute or two to figure out if it's their problem or not.
>
> Computer time is free, human time is not.  Computers are continually
> getting faster, while humans get slower.

I don't disagree that knowing exactly what revision caused the failure  
would be useful, it's just that I don't think it's useful enough for  
me to spend any time worrying about :-)

> Have you had a chance to play with git bisect yet?  It makes me feel
> squishy in my happy place.

Not yet, I don't usually bother tracking down an offending commit  
unless I'm particularly interested in what the hell they were thinking  
or if I need to discuss the whys and hows.

I'm mostly still using git as I was using svn except for heavy use of  
stash, branch and reset.

[Adam Heath-2]

Scott Gray wrote:

> On 6/01/2010, at 10:50 AM, Adam Heath wrote:
>
>> Scott Gray wrote:
>>> On 6/01/2010, at 10:19 AM, Adam Heath wrote:
>>>
>>>> Additionally, it'd be nice if once a failure is detected, it'd either
>>>> do an auto binary disection, to find the commit that actually failed
>>>> in the list of covered revisions, or it could just try them all.
>>>>
>>>> For instance, if 891234 is good, and then 892765 fails.  892765 would
>>>> get recorded as a state change, putting ofbiz into fail mode.  It
>>>> would then try 891562, 891840, 892496, and 892599, reporting that it
>>>> tried these other revisions, and noticed the error first occurred at
>>>> 892496, then ofbiz would stay in fail mode, and the buildbot would
>>>> keep quiet until it detects that ofbiz has been fixed.
>>>
>>> I have no idea if buildbot is capable of this but you're more than
>>> welcome to investigate it and request changes from infra.
>>
>> Standard open-source response, kinda expected it.  It'd still be nice
>> if the request was forwarded on, if it wasn't too much of a hassle.
>> If it is, I respect that, and I'll just add it to my infinite todo
>> list squared.
>
> The idea is to ask infra to do no more than is necessary (they're unpaid
> volunteers), so at the very least we (you) should find out if buildbot
> is capable of doing what you desire.
> As below human time isn't free and I'd prefer it if people deal with
> infra directly if the change is important enough to them.
>
>>> In general though I think it's usually fairly obvious what the cause of
>>> a problem is based on a quick look at the stdio from the failed build
>>> task.  I don't think it's a huge ask for each committer on the blamelist
>>> to take a minute or two to figure out if it's their problem or not.
>>
>> Computer time is free, human time is not.  Computers are continually
>> getting faster, while humans get slower.
>
> I don't disagree that knowing exactly what revision caused the failure
> would be useful, it's just that I don't think it's useful enough for me
> to spend any time worrying about :-)
>
>> Have you had a chance to play with git bisect yet?  It makes me feel
>> squishy in my happy place.
>
> Not yet, I don't usually bother tracking down an offending commit unless
> I'm particularly interested in what the hell they were thinking or if I
> need to discuss the whys and hows.
>
> I'm mostly still using git as I was using svn except for heavy use of
> stash, branch and reset.

I had to learn it quick.  We were attempting to manage a website, on
multiple internal machines, a staging server, a customer-hosted
production server, and a customer-hosted development server.  The size
of the website was originally 22G, but grew during development to
40G(it had lots and lots of videos).

Originally it was all managed in svn, but multi-server development
sucks.  We then moved to mercurial.  However, mercurial is buggy with
large systems, because it has a built-in limit due to its use of
python-strings internally.  In essence, a changeset can't be larger
than 2G.  If you don't update the site for a couple of days, it could
be entirely possible that the whole set of changes would be more than
2G, and it would just stop working.  Not to mention when files get
renamed, that mercurial would end up storing multiple copies of the
same blob of bytes(svn is better in that regard).

We finally settled on git.  During this time, we really only used it
like svn, but committing locally, then made use of it as an advanced
rsync deployment system.  The switch to git for this large website
happened last March.

Since then, I've become ecstatic with git.  It's not been a completely
new workflow; I've used svk ages ago(but not really that much, just
kinda like an offline commit), and mercurial 1.5 years ago.

>